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of  IEEE/RSJ  International  Conf.  on  Intelligent  Robots  and  Systems,  Oct.  11-15,  2009. 

B  .  Xu,  D.  J.  Stilwell,  A.  Gadre  and  A.J.  Kurdila,  “Analysis  of  local  observability  for  featurelocalization  in  a  maritime  environment  using 
an  omnidirectional  camera”,  Proc.  oflEEE/RSJ  International  Conf.  on  Intelligent  Robots  and  Systems,  pp.3666-3671,  Oct.  29-Nov.l,  2007. 

Aug.  2009- Jul  2010 

R.  Baraniuk,  “Randomized  Dimensionality  Reduction:  A  New  Framework  for  Signal  Processing  and  Communications,”  IEEE  International 
Symposium  on  Information  Theory  (ISIT),  Seoul,  Korea,  2009. 

P.  Binev,  “On  the  Greedy  Approach  to  Reduced  Basis  Method,”  I  Jaen  Conference  on  Approximation,  Ubeda,  Jaen,  Spain,  July  4  -  9,  2010 

P.  Binev,  “Nonlinear  Approximation  of  Surfaces,”  Seventh  International  Conference  on  Curves  and  Surfaces,  Avignon,  France,  June  24  - 
30,  2010 

P.  Binev,  “Sparse  Occupancy  Trees,  Workshop  about  High  Dimensional  Problems  and  Solutions,”  Paris,  France,  June  21  -  22,  2010 

P.  Binev,  “3D  Reconstructions  of  Simulated  Frequency  Agile  Lidar  Sensed  Concentration  Fields,”  2010  DTRA/NSF  Algorithm  Workshop, 
Chapel  Hill,  NC,  June  21-24,  2010. 

P.  Binev,  “2.5  D  Reconstructions  of  Simulated  Frequency  Agile  LiDAR  Concentration  Estimates  from  a  Fixed  Location,”  2010 
DTRA/NSF 

Algorithm  Workshop,  Chapel  Hill,  NC,  June  21-24,  2010. 

P.  Binev,  “Adaptive  Approximation  of  Surfaces,”  International  Conference  on  Constructive  Theory  of  Functions,  Sozopol,  Bulgaria,  June 
3-9,  2010 


P.  Binev,  “Adaptive  Approximation  of  Surfaces,”  Thirteenth  International  Conference  in  Approximation  Theory,  San  Antonio,  Texas, 
March,  6  -  10,  2010. 

P.  Binev,  “Nonlinear  Processing  of  HAADF  STEM  Data,”  Imaging  in  Electron  Microscopy  II,  Columbia,  SC  February  18  -  23,  2010. 

R.  Sharpley,  “Segmentation  and  Compression  of  Heterogeneously  Distributed  3D  Point  Clouds,”  3D  Modelling  of  Urban  Terrain  -  NATO 
SET  118,  Ettlingen,  Germany,  November  3-5,  2009. 

P.  Binev,  “Sparsity  in  Adaptive  Approximation,  Nonlinear  and  Adaptive  Approximation,”  Castle  Reisensburg  (Giinzburg),  Germany, 
September  30  -October  3,  2009. 

Luke  Owens,  “An  Algorithm  for  Surface  Encoding  and  Reconstruction  From  3D  Point  Cloud  Data,”  Numerical  Analysis  Seminar,  Texas 
A&M,  April  2010. 

David  Jimenez,  “Matching  of  Point  Configurations:  An  Approach  Through  Grammians,”  Approximation  Theory  Conference,  San  Antonio, 
March  2010. 

Guergana  Petrova,  “Instance-optimality  in  probability  with  an  11 -minimization  decoder,”  Curves  and  Surfaces  Conference,  June,  2010. 

R.  Tsai,  R.  Takei,  Y.  Landa,  and  N.  Tanushev,  "A  few  optimal  path  planning  problems  and  reversible  cars  with  constrained  turning  radii”, 
AFOSR  Workshop  on  Computational  Control,  Nov  2009. 

R.  Tsai,  Y.  Landa,  N.  Tanushev,  Y.  Li,  and  S.  Osher,  "Inverse  source  problems  in  complicated  domains,”  McGill  University,  Canada,  Jan 

2010. 

R.  Tsai  and  R.  Takei,  "Optimal  trajectories  for  curvature  constrained  motions",  Bar-Ilan  University,  Israel,  May  2010. 

R.  Tsai,  Y.  Landa,  N.  Tanushev,  Y.  Li,  and  S.  Osher,  "Inverse  source  problems  in  domains  with  holes,”  Technion  University,  Israel.  June 

2010. 

R.  Tsai,  Y.  Landa,  N.  Tanushev,  Y.  Li,  and  S.  Osher,  "Inverse  source  problems  in  domains  with  holes,"  NSF/DTRA  Algorithm  Workshop, 
June  2010. 

8/2010-7/2011 

P.  Binev,  "Localized  Nonlocal  Means  with  Application  to  Electron  Microscopy" 

Workshop  on  Wavelet  and  Multiscale  Methods,  Oberwolfach,  Germany,  August  1  -  7,  2010. 

P.  Binev,  "High-Dimensional  Scattered  Data  Approximation  with  Sparse  Occupancy  Trees",  Workshop  on  Wavelet  and  Multiscale 
Methods,  Oberwolfach,  Germany,  August  1  -  7,  2010. 

P.  Binev,  "Greedy  algorithms  for  the  Reduced  Basis  Method",  Multivariate  Approximation  and  Interpolation  with  Applications,  Edinburgh, 
UK 

September  6  -  10,  2010. 

R.  Sharpley,  "Mathematical  methods  for  high  resolution  image  formation  from  scanning  transmission  electron  microscopy" 

The  Second  International  Conference  on  NUMERICAL  ANALYSIS  AND  OPTIMIZATION,  Muscat,  OMAN,  January  5,  201 1. 

P.  Lamby,  "Total  Variation  Regularization  in  Tomography-  Part  I" 

New  Frontiers  in  Imaging  and  Sensing  Seminar  Series,  Columbia,  SC 
January  28,  2011. 

P.  Lamby,  "Total  Variation  Regularization  in  Tomography-  Part  II" 

New  Frontiers  in  Imaging  and  Sensing  Seminar  Series,  Columbia,  SC 
February  4,  201 1. 


D.  Savu,  "A  Basic  Concept  in  Compressive  Sensing  Applied  to  a  STEM  Image  Reconstruction  Problem"  Workshop-  New  Frontiers  in 
Imaging  and  Sensing,  Columbia,  SC,  February  17  -  22,  201 1. 


P.  Binev,  "High  Quality  Image  Formation  by  Unconventional  Data  Acquisition  in  STEM"  Workshop-  New  Frontiers  in  Imaging  and 
Sensing,  Columbia,  SC 
February  17-22,2011. 

P.  Lamby,  "TV-Regularized  Algebraic  Reconstruction  for  Limited-Angle  Tomography"  Workshop-  New  Frontiers  in  Imaging  and  Sensing, 
Columbia,  SC 

February  17  -  22,  201 1  B.  Karaivanov,  "Polygonal  representation  of  3D  urban  terrain  point-cloud  data-  Part  I"  New  Frontiers  in  Imaging 
and  Sensing  Seminar  Series,  Columbia,  SC  February  23,  2011. 

B.  Karaivanov,  "Polygonal  representation  of  3D  urban  terrain  point-cloud  data-  Part  II"  New  Frontiers  in  Imaging  and  Sensing  Seminar 
Series,  Columbia,  SC,  March  3,  201 1. 

P.  Binev,  "Classification  using  nonlinear  approximation,”  International  Symposium  on  Approximation  Theory,  Nashville,  TN,  May  17  - 

21,2011. 

P.  Binev,  "Point  cloud  processing  using  reliable  sets,”  2011  Algorithm  Workshop,  Boston,  MA,  June  7  -  9,  201 1. 

P.  Binev,  "Sparse  Occupancy  Trees  for  Approximation  and  Classification,”  Workshop  on  Theoretical  Aspects  of  High-Dimensional 
Problems  and  Information-  Based  Complexity,  Bonn,  Germany,  June  20  -  24,  2011 

P.  Binev,  "Advanced  Data  Acquisition  in  Electron  Microscopy,”  FoCM  2011  Workshop  on  Approximation  Theory,  Budapest,  Hungary, 
July  8-  10,  2011 

P.  Binev,  "Advanced  Image  Formation  in  Electron  Microscopy," 

Aachen  Conference  on  Computational  Engineering  Science,  Aachen,  Germany 
July  13 -15,  2011 

P.  Binev,  "Sparse  Tree  Approximation  in  High  Dimensions,”  Applied  Harmonic  Analysis  and  Multiscale  Computing,  Edmonton,  Canada, 
July  25  -28,  2011. 

H.  Zhao,  "Shape  Reconstruction  from  Point  Cloud,”  Dagstuhl-Seminar  on  ’’Innovations  for  Shape  Analysis:  Models  and 
Algorithms" Schloss  Dagstuhl  -  Leibniz-  Zentrum  fur  Informatik,  Germany. 

April  201 1,  H.  Zhao,  "Analysis  and  Segmentation  of  Point  Cloud”, 

International  Workshop  on  Image  Processing,  Computer  Vision,  Compressive  Sensing  and  Related  Applications,  Seoul,  KOREA, 
December  2010. 

R.  DeVore,  "Greedy  Algorithms  for  High  Dimensional  Approximation," 

Foundations  of  Computa-  tional  Mathematics  Conference,  Budapest,  Hungary, 

July  4-14,  2011 

R.  DeVore,  "Greedy  Algorithms  for  Generating  Reduced  Bases," 

Aachen  Conference  on  Computational  Engineering  Science,  Aachen,  Germany 
July  13-15,2011. 

R.  DeVore,  "Approximation  Theory,"  II  Jaen  Conference  on  Approximation  Theory,  Ubeda,  Jaen,  Spain,  June  26  -  July  1,  2011. 

R.  DeVore,  "Reduced  Basis,”  Reduced  Basis  Methods  in  High  Dimensions  Workshop,  Paris,  France,  June  23-24,  2011. 

8/2011-7/2012 

R.  Baraniuk,  “Randomized  Dimensionality  Reduction  and  Compressive  Sampling,”  Padovani  Lecture,  Information  Theory  School,  IEEE 
Information  Theory  Society,  Austin,  2011. 


R.  Baraniuk,  “Compressive  Sensing  and  Signal  Processing,”  University  of  Delaware  Distinguished  Lecture  Series,  2011. 


"On  Adaptive  Strategies  in  Finite  Element  Methods",  Computational  Methods  in  Applied  Mathematics  (CMAM2012)  Berlin,  Germany, 
July  30  -  August  3,  2012. 

"On  the  Greedy  Approach  to  the  Reduced  Basis  Method",  Workshop  on  Model  Order  Reduction  in  PDE  Constrained  Optimization, 
Hamburg,  Germany,  July  25  -  27,  2012. 

"Adaptive  Partitioning  Algorithms  for  Classification",  Eighth  International  Conference  on  Mathematical  Methods  for  Curves  and  Surfaces, 
Oslo,  Norway,  June  28  -  July  3,  2012. 

"Classification  Algorithms  using  Adaptive  Partitioning",  Institute  for  Geometry  and  Practical  Mathematics,  RWTH  -  Aachen,  Aachen, 
Germany,  July  12,  2012. 

"Greedy  Approach  to  Reduced  Basis  Method",  University  of  Hawaii,  Honolulu,  March  7,  2012. 

“Manifold-based  Signal  Separation  and  Recovery,”  International  Workshop  on  Signal  and  Image  Geometry  Modelling  and  Approximation 
(SIGMA),  Marseille,  France. 

“Open  Education  for  Disaster  Preparation,”  FLASH  2012  Annual  Conference,  Orlando. 

“Compressive  Sensing:  8  Years  After,”  46th  Asilomar  Conference  on  Signals,  Systems,  and  Computers,  Pacific  Grove,  CA 

“Optimization  based  Sparse  Signal  Recovery,”  21st  International  Symposium  on  Mathematical  Programming  (ISMP  2012),  Berlin 

“Compressive  Signal  Processing,”  Center  for  Advanced  Signal  and  Image  Science  (CASIS)  Workshop,  Lawrence  Livermore  National 
Laboratory 

“Compressive  Signal  Processing,”  Mohammed  Dahleh  Distinguished  Lecture,  UCSB 
"Greedy  Algorithms  in  Banach  Spaces",  Paris,  France,  June  2012 

"Lecture  1:  Encoding  Signals  and  Lecture  2:  Image  Compression",  AICES  Lectures,  Aachen,  Germnay,  June  2012 
"Greedy  Algorithms  in  Banach  Spaces",  Davinci  Lecture  Series,  Milan,  Italy,  July  2012 
"Greedy  Algorithms  in  Banach  Spaces",  Departmental  Series,  Aachen,  Germany,  July  2012 

’’Geometric  Understanding  of  Point  Clouds  Using  Laplace-Beltrami  Operator”,  Advances  in  Scientific  Computing,  Imaging  Science  and 
Optimization,  in  honor  of  Stanley  Osher’s  70th  birthday,  IP  AM,  UCLA,  4/4-4/6,  2012. 

’’Solving  partial  differential  equations  on  point  clouds”,  The  International  Confer-  ence  on  Scientific  Computing,  in  honor  of  Tony  Chan’s 
60th  birthday,  Hong  Kong,  1/4- 1/7,  2012. 


H.  Zhao,  "Geometric  Understanding  of  Point  Clouds  Using  Laplace-Beltrami  Opera¬ 
tor",  Advances  in  Scientic  Computing,  Imaging  Science  and  Optimization, 

IP  AM,  UCLA,  4/4-4/6,  2012. 

H.  Zhao,  "Solving  partial  di  erential  equations  on  point  clouds",  The  International  Conference  on  Scienti  c  Computing,  Hong  Kong, 
1/4- 1/7,  2012. 

H.  Zhao,  "Shape  Reconstruction  from  Point  Cloud",  Dagstuhl-Seminar  on  "Innovations 
for  Shape  Analysis:  Models  and  Algorithms",  4/2011,  Schloss  Dagstuhl  - 
Leibniz-Zentrum  fur  Informatik,  Germany. 


H.  Zhao,  "Analysis  and  Segmentation  of  Point  Cloud”,  International  Workshop  on  Im¬ 
age  Processing,  Computer  Vision,  Compressive  sensing  and  Related  Applica¬ 
tions,  12/2010,  Seoul,  Korea. 

R.  DeVore,  Greedy  Algorithms  in  Banach  Spaces,  Davinci  Lecture  Series,  Milan,  Italy,  July  2012. 

R.  DeVore,  Greedy  Algorithms  in  Banach  Spaces,  Departmental  Series,  Aachen,  Germany  July  2012. 

R.  DeVore,  Greedy  Algorithms  in  Banach  Spaces,  Paris,  France  June  2012. 

R.  DeVore,  Lecture  1:  Encodingi  Signals  and  Lecture  2:  Image  Compression,  AICES  Lectures,  Aachen,  Germany,  June  2012. 

D.  Petrova,  Sofia  University,  June  2012. 

8/2012-1/2013 

R.  Baraniuk,  “Video  Compressive  Sensing,”  Matheon  Workshop  on  Compressed  Sensing  and  its  Applications,  Berlin,  2013. 

R.  Baraniuk,  “Convex  Optimization  for  Learning  Near-Isometric  Linear  Embeddings,”  GlobalSIP  2013  Symposium  on  Low-Dimensional 
Models  and  Optimization  in  Signal  Processing,  Austin,  TX,  2013 

R.  Baraniuk,  “Sparsity  and  Signal  Processing,”  SPARS  Workshop,  EPFL,  Lausanne,  Switzerland 
“Video  Compressive  Sensing,”  BASP  Frontiers  Workshop,  Switzerland,  2013. 

Number  of  Presentations:  107.00 

Non  Peer-Reviewed  Conference  Proceeding  publications  (other  than  abstracts): 


Received  Paper 


TOTAL: 


Number  of  Non  Peer-Reviewed  Conference  Proceeding  publications  (other  than  abstracts): 


Peer-Reviewed  Conference  Proceeding  publications  (other  than  abstracts): 


Received 


05/13/2013  24.00  J.  P.  Slavinsky,  Jason  N.  Laska,  Mark  A.  Davenport,  Richard  G.  Baraniuk.  The  compressive  multiplexer 
for  multi-channel  compressive  sensing, 

ICASSP  2011  -  2011  IEEE  International  Conference  on  Acoustics,  Speech  and  Signal  Processing 
(ICASSP).  2011/05/22  00:00:00,  Prague,  Czech  Republic.  :  , 

08/31/201 1  33.00  Sang-Mook  Lee,  Jeong  Joon  Im,  Bo-Hee  Lee,  Alexander  Leonessa,  Andrew  Kurdila.  A  real-time  grid  map 

generation  and  object  classification  for  ground-based  3D  LIDAR  data  using  image  analysis  techniques, 
2010  17th  IEEE  International  Conference  on  Image  Processing  (ICIP  2010).  2010/09/26  00:00:00,  Hong 
Kong,  Hong  Kong.  :  , 

08/31/201 1  34.00  Bin  Xu,  Daniel  J  Stilwell,  Andrew  J  Kurdila.  A  receding  horizon  controller  for  motion  planning  in  the 

presence  of  moving  obstacles, 

2010  IEEE  International  Conference  on  Robotics  and  Automation  (ICRA  2010).  2010/05/03  00:00:00, 
Anchorage,  AK.  :  , 

TOTAL:  3 


Number  of  Peer-Reviewed  Conference  Proceeding  publications  (other  than  abstracts): 


(d)  Manuscripts 


Received 


Paper 


05/13/2013 

05/13/2013 

05/13/2013 

09/23/2008 

09/23/2008 

11/30/2009 

11/30/2009 

11/30/2009 

12/01/2009 

12/01/2009 

12/20/2009 

12/20/2009 

12/20/2009 

TOTAL: 


48.00  Jian  Liang,  Frederick  Park,  Hongkai  Zhao.  Robust  and  Efficient  Implicit  Surface  Reconstruction  for  Point 
Clouds  Based  on  Convexified  Image  Segmentation, 

J  Sci  Comput  (03  2012) 

46.00  Rongjie  Lai,  Jiang  Liang,  Hongkai  Zhao.  A  Local  Mesh  Method  for  Solving  PDES  on  Point  Clouds, 

Inverse  Problems  and  Imaging  (01  2012) 

43.00  Songting  Luo,  Leonidas  Guibas,  Hong-Kai  Zhao.  Euclidean  skeletons  using  closest  points, 

Inverse  Problems  and  Imaging  (02  2011) 

1.00  .  Anisotropic  Smoothness  Spaces  via  Level  Sets, 

0 

2.00  Yanina  Landa,  Richard  Tsai.  Visibility  of  Point  Clouds  and  Exploratory  Path  Planning  in  Unknown 
Environments, 

0 

3.00  Yanina  Landa,  Haochong  Shen,  Ryo  Takei,  and  Yen-Hsi  R.  Tsai.  Autonomous  Source  Discovery  and 
Navigation  in  Complicated  Environments, 

IEEE  International  Conference  on  Robotics  and  Automation  (11  2009) 

5.00  Martin  Burger,  Yanina  Landa,  Nicolay  M.  Tanushev,  Richard  Tsai.  Discovering  a  Point  Source  in 
Unknown  Environments, 

Eighth  International  Workshop  on  the  Algorithmic  Foundations  of  Robotics  (01  2008) 

6.00  Ryo  Takei,  Richard  Tsai,  Haochong  Shen,  Yanina  Landa.  A  Practical  Path-planning  Algorithm  for  a 
Vehicle  with  a  Constrained  Turning  Radius:  a  Hamilton-Jacobi  Approach, 

Automatic  Control  (1 1  2009) 

7.00  E.  Castillo,  H.  Zhao.  Point  Cloud  Segmentation  via  Constrained  Nonlinear  Least  Squares  Surface  Normal 
Estimates, 

Twenty-Third  IEEE  Computer  Society  Conference  on  Computer  Vision  and  Pattern  Recognition  (12  2009) 

8.00  Y.  Xi,  Y.  Duana,  H.  Zhao.  A  Nonparametric  Approach  for  Noisy  Point  Data  Preprocessing, 

International  Journal  of  CAD/CAM  (12  2009) 

10.00  J.  N.  Laska,  P.  T.  Boufounos,  M.  A.  Davenport,  R.  G.  Baraniuk.  Democracy  in  Action:  Quantization, 
Saturation,  and  Compressive  Sensing, 

IEEE  Transactions  on  Signal  Processing  (12  2009) 

11.00  R.  G.  Baraniuk,  V.  Cevher,  M.  F.  Duarte,  C.  Hegde.  Model-Based  Compressive  Sensing, 

IEEE  Transactions  in  Information  Theory  (12  2009) 

12.00  C.  Hegde,  M  F.  Duarte,  V.  Cevher.  Compressive  Sensing  Recovery  of  Spike  Trains  Using  A  Structured 
Sparsity  Model, 

Proc.  Workshop  on  Signal  Processing  with  Adaptive  Sparse  Representations  (SPARS)  (12  2009) 
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Number  of  Manuscripts: 


Books 


Received 


TOTAL: 


Patents  Submitted 


Patents  Awarded 


Awards 

Richard  Baraniuk,  SPIE  Compressive  Sampling  Pioneer  Award,  2012. 

Richard  Baraniuk,  IEEE  Signal  Processing  Educator  Award,  2010. 

Ronald  DeVore,  Gold  Medal  of  the  French  Mathematical  Society,  2011. 

Stanley  Osher,  Fellow  of  the  Society  of  Industrial  and  Appied  Mathematics,  2009. 
Stanley  Osher,  Elected  to  the  American  Academy  of  Arts  and  Sciences,  2009. 

Stanley  Osher,  Honorary  Doctoral  Degree,  Hong  Kong  Baptict  University,  2009. 

Stanley  Osher,  Plenary  Speaker,  International  Conference  of  Mathematicians,  2010. 
Stanley  Osher,  John  von  Neumann  Lecture  for  the  2013  SIAM  Annual  Meeting,  2013. 
Stanley  Osher,  Fellow  of  the  American  Mathematical  Society,  2012. 

Graduate  Students 


NAME 

PERCENT  SUPPORTED  Discipline 

Aditya  Kumar 

0.04 

Alexander  Mamonov 

0.14 

AN  Ayremlou 

0.04 

AN  Haddad 

0.09 

AN  Mousavi 

0.04 

Alok  Sood 

0.03 

Aman  Jain 

0.09 

Andrew  Temylakov 

0.08 

Andrew  Waters 

0.03 

Ben  Smith 

0.10 

Billy  Clack 

0.05 

Bin  Dong 

0.15 

Bin  Xu 

0.10 

Chinmay  Hegde 

0.03 

Daniel  Calderon 

0.04 

Daniel  Savu 

0.04 

Daniel  Soares 

0.02 

Dharmpal  Takhar 

0.02 

Edward  Castillo 

0.13 

Francisco  Blanco-Silva 

0.09 

Gil  Ariel 

0.12 

Guanchun  Wang 

0.26 

Haipeng  Zheng 

0.12 

Haochong  Shen 

0.03 

Huiyi  Hui 

0.06 

Igor  Yanovsky 

0.11 

Jason  Laska 

0.05 

Jeong  Im 

0.00 

Jian  Liang 

0.25 

Jieqi  Yu 

0.27 

John  Freeman 

0.02 

Kamala  Diefenthaler 

0.02 

Manjari  Narayan 

0.10 

Marco  Duarte 

0.01 

Mark  Davenport 

0.02 

Matthew  Moravec 

0.05 

Matthew  Summers 

0.09 

Michael  Bennet 

0.22 

Mona  Sheikh 

0.09 

Moritz  Allmaras 

0.06 

Nicolay  Tanushev 

0.14 

Petros  Boufounos 

0.05 

Philipp  Lamby 

0.11 

Seong  Jun  Kim 

0.04 

Shang  Shang 

0.03 

Shriram  Sarvotham 

0.03 

Teng  Wang 

0.04 

Ting  Sun 

0.04 

Volkan  Cevher 

0.03 

Woon  Kim 

0.10 

Xinwei  Zhang 

0.11 

Yingying  Li 

0.02 

Yu  Lei 

0.14 

Yue  Li 

0.02 

Zhuo  Zhang 

0.09 

FTE  Equivalent: 

4.29 

Total  Number: 

55 

Names  of  Post  Doctorates 


NAME 

PERCENT  SUPPORTED 

Alexander  Mamonov 

0.28 

Aswin  Sankaranarayanan 

0.10 

Colin  Macdonald 

0.01 

Edward  Castillo 

0.44 

Frederic  Park 

0.58 

Gil  Ariel 

0.12 

Nicolay  Tanushev 

0.14 

Vahram  Stepanyan 

0.01 

Yanina  Landa 

0.38 

FTE  Equivalent: 

2.06 

Total  Number: 

9 

Names  of  Faculty  Supported 


NAME 

PERCENT  SUPPORTED 

National  Academy  Member 

Andrew  Kurdila 

0.09 

Guergana  Petrova 

0.17 

Hongkai  Zhao 

0.08 

Mark  Pierson 

0.03 

Michael  Roan 

0.02 

No 

Peter  Binev 

0.09 

Richard  Baraniuk 

0.00 

Richard  Tsai 

0.19 

Robert  Sharpley 

0.11 

Ronald  DeVore 

0.03 

Sanjeev  Kulkarni 

0.08 

Stanley  Osher 

0.13 

Yes 

Wolfgang  Dahmen 

0.02 

FTE  Equivalent: 

1.04 

Total  Number: 

13 

Names  of  Under  Graduate  students  supported 


NAME 

PERCENT  SUPPORTED 

Discipline 

Josiah  Manson 

0.11 

computer  science 

Peter  Hall 

0.05 

mathematics 

Matthew  Madison 

0.04 

mathematics 

Randy  Ransom 

0.11 

computer  science 
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Overview 

This  report  summarizes  efforts  and  accomplishments  of  the  eight  universities  collaborating  on 
Multi-University  Research  Initiative  of  the  Army  Research  Office  (ARO),  grant  W911NF-07-1- 
0185,  entitled  Model  Classes,  Approximation,  and  Metrics  for  Dynamic  Processing  of  Urban 
Terrain  Data.  This  project  has  developed  theory,  algorithms,  software  and  experiments  for  the 
synthesis  of  urban  terrain  maps  from  dynamic  point  cloud  sensor  data. 

The  main  goals  of  the  research  team  have  been  (i)  to  capture  high  order  topology  through  implicit 
representation  of  surfaces,  (ii)  to  develop  multiscale  and  adaptive  algorithms  which  enable  various 
resolutions  of  the  rendered  surface  governed  by  the  local  density  of  the  point  clouds,  (iii)  to  derive 
algorithms  for  fast  computation  of  signed  distances  to  the  terrain  surface  thereby  giving  field  of 
view  from  specified  observation  points,  (iv)  to  derive  fast  and  efficient  methodologies  for  the  use 
of  dynamic  point  cloud  measurements  for  the  navigation  and  control  of  autonomous  vehicles  in 
three  dimensions,  and  (v)  to  develop  change  detection  theory  and  methods  that  employ  point  cloud 
observations  taken  at  different  times. 
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l.  Compression  of  Point  Clouds 

Terrain  point  cloud  data  are  typically  acquired  through  some  form  of  LiDAR  sensing  or  from 
post-processing  CCD  image  sequences  via  feature  point  tracking  algorithms.  They  form  a  rich  re¬ 
source  that  is  important  in  a  variety  of  applications  including  navigation,  line  of  sight  calculations, 
and  terrain  visualization.  The  point  clouds  themselves  are  too  cumbersome  and  large  to  be  used 
directly  for  these  purposes.  They  need  to  be  converted  to  a  simpler  platform  that  is  more  efficient 
and  still  contains  all  of  the  features  of  the  terrain,  present  in  the  point  cloud,  that  are  needed  for 
these  applications.  A  naive  approach  would  be  to  take  local  averages  of  data  heights  to  obtain  pixel 
intensities  (and  therefore  a  pixelized  image)  and  then  employ  the  techniques  of  image  processing 
to  make  a  conversion  into  a  wavelet  or  other  multiscale  representation.  However,  this  approach 
is  not  successful  for  several  reasons.  Foremost  among  these  is  that  terrains  are  not  images.  They 
have  certain  topology  and  geometry  that  must  be  extracted  and  preserved.  A  second  related  point 
is  that  the  usual  least  squares  metrics  used  in  image  processing  do  not  match  the  intended  appli¬ 
cations  for  terrain  maps.  For  example,  capturing  long  thin  structures  such  as  poles,  towers,  and 
wires  are  essential  for  navigation,  but  are  not  given  priority  in  PSNR  metrics  employed  for  images. 
Another  issue  is  that  terrain  point  clouds  usually  have  missing  data,  occlusions  and  noise  which 
do  not  appear  in  many  other  applications.  While  surface  reconstruction  from  point  clouds  is  now 
a  dominant  theme  in  computer  graphics,  very  little  of  this  work  addresses  terrain  data  per  se.  Of 
course,  one  could  argue  that  one  can  simply  apply  one  of  the  vast  number  of  surface  processing 
algorithms  in  computer  graphics.  However,  these  algorithms  are  typically  built  for  high  resolution 
data  which  is  not  the  case  for  general  terrain  data,  which  suffers  from  occlusions,  missing  data, 
noise,  and  variable  resolution. 

Our  platform  for  urban/terrain  processing  is  built  on  the  following  principles: 

•  Distortion  is  measured  in  the  Hausdorff  metric,  which  we  argue  is  a  good  match  for  the 
application  demands. 

•  The  decomposition  is  organized  in  a  multiscale  octree  giving  coarse  to  fine  resolution. 

•  Each  node  of  the  tree  corresponds  to  a  dyadic  cube  and  is  equipped  with  a  low  degree 
polynomial  fit  to  the  point  cloud  on  this  cube  that  minimizes  the  local  Hausdorff  error  of 
this  fit. 

•  The  tree  is  organized  into  subtrees  each  of  which  corresponds  to  a  certain  accuracy  of 
resolution  in  the  Hausdorff  metric. 

•  The  tree  and  nodal  information  can  be  efficiently  encoded  using  predictive  encoding. 

•  Upon  receiving  the  tree  and  nodal  information,  the  user  can  easily  convert  this  information 
to  a  format  that  matches  the  intended  application. 

•  Primitives  such  as  normals,  curvature,  and  other  information  can  be  extracted  from  the  tree 
and  nodal  information. 

This  may  be  the  only  terrain  point  cloud  encoder  that  gives  rate/distortion  performance  and  the 
only  encoder  using  the  Hausdorff  metric.  There  are  no  other  studies  known  to  us  with  which 
we  could  compare  our  work.  This  lack  of  documented  encoding  algorithms  has  been  one  of  the 
drawbacks  in  the  development  of  terrain  point  cloud  encoding.  The  MURI  encoder,  as  described 
in  [26],  gives  a  benchmark  to  which  other  researchers  can  now  compare  their  work. 


1.1.  Targeted  Applications  Guiding  Algorithm  Development.  To  demonstrate  the  applicability 
of  the  techniques  being  developed  in  this  project,  we  have  focused  on  several  targeted  applications 
that  we  now  describe. 


4 


Figure  1 .  Two  reconstructions  (left  and  middle)  from  point  cloud  scans  generated 
from  separate  flyovers  of  a  simulated  aerial  vehicle.  The  combined  surface  (right) 
is  colored  according  to  the  distance  between  the  surfaces.  Blue  indicates  little  to  no 
difference  while  green  and  red  indicate  larger  differences  in  the  surfaces. 


Change  Detection:  We  have  consulted  with  ARL  researchers  at  Adelphi  and  AFRL  sensors  direc¬ 
torate  personnel  at  Wright  Patterson  to  formulate  a  problem  which  illustrates  the  potential  of  the 
algorithms,  while  addressing  the  RCAs  of  the  MURI  effort.  A  timely  and  relevant  problem  which 
satisfies  these  criteria  is  the  problem  of  change  detection  in  geometric  scenes  imaged  by  fixed  and 
mobile  surveillance  platforms  equipped  with  various  sensor  packages.  Given  two  sets  of  point 
cloud  data  of  a  given  scene,  generated  at  different  times,  we  would  like  to  identify  any  changes 
that  have  taken  place  in  the  scene.  The  ultimate  goal  of  this  application  is  to  have  algorithms  which 
can  be  applied  to  point  cloud  data  generated  in  real  time  by  a  battery  of  navigating  sensors. 

We  have  incorporated  our  encoder,  together  with  various  denoising  and  enhancement  routines 
in  order  to  demonstrate  a  proof  of  concept.  We  have  developed  simulated  LIDAR  point  clouds  of 
scenes  which  share  some  common  regions,  incorporated  changes  in  these  scenes  and  have  run  our 
algorithms  on  each  scene  to  see  if  it  can  identify  the  changes.  To  compute  changes,  we  measure 
the  Hausdorff  distance  between  the  two  reconstructed  surfaces  we  generate.  Figure  1  shows  the 
performance  of  our  algorithm  on  a  typical  example  where  the  one  point  cloud  is  of  a  scene  with  a 
tank  sitting  in  an  alley  between  two  buildings. 

Path  Planning:  Next,  we  have  collaborated  with  researchers  from  UT  and  USC  and  performed 
several  experiments  using  the  USC  Simulator,  TAMU  Surface  Reconstruction  methods  and  UT 
path  planning  algorithms  to  fully  navigate  autonomous  vehicles.  In  our  tests,  a  simulated  LiDAR 
device  was  positioned  and  a  scan  of  the  environment  performed.  The  resulting  scan  data  was  used 
to  reconstruct  the  visible  surfaces.  From  the  approximated  surface  and  sensor  location  a  visibility 
volume  was  produced  and  processed  by  the  UT  path  planner  in  order  to  move  the  sensor  to  its  next 
position,  see  Figure  2. 

Low  Power  Communication:  We  acknowledge  that  many  defense  missions  require  real-time 
visualization  of  reconstructed  surfaces  on  low  power  bandwidth-limited  devices.  Our  encoding 
scheme  allows  for  such  devices  to  receive  encoded  surfaces  in  a  progressive  streaming  fashion. 
The  TAMU  group  has  developed  and  demonstrated  that  these  encoded  surfaces  can  be  decoded 
and  reconstructed  for  visualization  on  current  consumer  tablet  and  smart  phone  technology,  see 
Figure  3. 

LIDAR  Simulation:  One  of  the  difficulties  in  testing  any  algorithm  is  the  availability  of  suitable 
data.  Acquisition  of  real  data  can  not  only  be  time  consuming  but  may  also  be  too  costly  and 
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Figure  2.  An  example  set  of  three  aggregate  reconstructions  from  a  closed-loop 
simulation  using  the  UT  path  planner  and  TAMU  Surface  Reconstruction.  Each 
reconstruction  shows  a  single  scan,  colored  by  height,  that  was  used  to  update  the 
reconstruction. 


Figure  3.  A  snapshot  of  the  TAMU  Surface  Viewer  running  on  an  iPad.  The 
surface  shown  has  been  decoded  and  reconstructed  locally  on  the  iPad  for  display. 

perhaps  impossible  with  available  sensors  and  vehicles.  To  address  this,  as  a  part  of  our  outreach 
program,  we  have  developed  an  editor  for  3D  point  clouds  that  can  work  with  data  produced  by 
the  USC  Simulator  in  order  to  construct  point  clouds  suitable  for  testing.  The  point  cloud  editor 
supports  basic  operations  such  as  cut,  trim,  copy  and  paste  along  with  more  advanced  operations 
including  denoising.  The  editor  is  publicly  available  as  part  of  the  latest  PCL  (pointclouds.org) 
source  release.  In  addition,  Matthew  Hielsberg,  a  researcher  at  TAMU,  has  coordinated  with  PCL 
staff  and  the  PCL  community  to  identify  a  set  of  existing  repositories  for  both  point  cloud  data 
and  point  cloud  processing  packages.  As  a  result  of  this  collaboration  the  PCL  website  will  now 
house  a  permanent  set  of  up-to-date  links  for  the  community  (http://pointclouds.org/media/),  see 
Figure  4. 

1.2.  Algorithm  Development.  Next,  we  will  describe  in  some  details  our  algorithms.  The  en¬ 
coder  assigns  bits  to  determine  the  occupancy  tree,  and  to  determine  each  of  the  polynomials  on 
the  terminal  leaves.  The  result  is  a  compression  of  the  original  point  cloud  into  a  succinct  progres¬ 
sive  bitstream.  Typical  compression  rates  are  from  100-1  up  to  500-1  depending  on  the  complexity 
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To  facilitate  the  open  testing  of  algorithms  the  PCL  community  recognizes  the  need  for  free  to  use 
data.  As  such  a  collection  of  open  data  sets  can  be  found  on  the  PCI  svn  repository.  The  files  in  this 
repository  are  stored  in  the  PCD  format,  which  can  be  easily  read  using  PCL  tools. 

In  addition  to  the  PCL  repository  there  are  many  other  data  repositories  containing  free  downloadable 
data  (in  various  formats).  The  following  is  a  list  of  such  sites  containing  links  to  data  that  is  free  to  use 
for  educational  work.  We  encourage  everyone  to  please  read  each  sites'  licensing  information  for  any 
data  used  and  appropriately  acknowledge  the  authors  in  any  publications  or  derived  work. 

If  you  wish  to  add  to  this  list  or  if  you  find  a  broken  link  feel  free  to  email  me  directly  fhielsberi. 


Data  Software 


Repositories 


•  3D  Point  Clouds 

O  Autodesk  Research  -  Digital  210  King 

Sample  scan  and  image  data  of  the  Autodesk  Research  office, 
o  Autonomous  Systems  Lab  (ETH) 

Laser  and  Kinect  data  from  structured  and  unstructured  environments.  This  group  has  a  list  of 
useful  related  links  as  well  as  a  similar  set  of  links  to  data. 
o  Canadian  Planetary  Emulation  Terrain  3D  Mapping  Dataset 

Laser  scans  of  unstructured  terrains  from  rover  platforms.  Includes  Matlab  code  for  data 
parsing. 

«  Leica  Geosvstems  f projected  data! 

Example  databases  of  laser  scan  data, 
o  libLAS  Samples 

Sample  data  sets  in  the  las  and  laz  formats. 


Sample  data  sets  for  pointools  evaluation. 

Pseudo-simulated  lidar  data  created  for  change  detection  experiments. 


Registered  3D  scans  of  indoor  and  outdoor  environments.  Some  data 
color  information. 


include  thermal  and 


o  SmartMultiMedia  LiDAR  Software  and  Point  Cloud  Sample  Data 

Sample  LiDAR  scan  data  sets,  including  links  to  commercial  point  cloud  software. 
•  Labeled  3D  Point  Clouds 

Fully  labeled  point  clouds  of  urban  scenes  around  the  CMU  campus. 


Figure  4.  A  snapshot  of  the  prototype  site  put  together  for  the  PCL  community. 
This  site  is  to  serve  as  a  central  location  for  the  storage  of  links  to  publicly  available 
data  and  software  for  the  processing  of  point  cloud  and  related  data. 


of  the  point  cloud.  At  these  compression  rates,  the  essential  geometry  and  topology  of  the  under¬ 
lying  terrain  surface  are  preserved  and  high  quality  reconstruction  is  possible  from  the  compressed 
file,  see  [26].  We  have  collaborated  with  USC  in  the  generation  of  the  occupancy  tree  and  with 
Rice  by  employing  their  predictive  tree  encoding  algorithms  to  enhance  the  compression  rates.  An 
alternative  to  the  Rice  encoder  was  proposed  in  [50],  where  a  compression  of  the  octree  structure 
together  with  the  linear  functions  at  each  of  its  nodes  is  presented.  The  encoding  of  a  linear  func¬ 
tion  at  a  node  is  performed  by  encoding  the  distances  from  three  points  associated  with  that  node 
to  the  linear  function  and  an  additional  bit.  The  reconstruction  is  done  by  solving  a  simple  linear 
optimization  problem  with  a  nonlinear  constraint. 

During  the  grant  period,  several  essential  difficulties  have  been  resolved  in  the  encoding  and 
display  algorithms.  In  the  encoding,  it  has  been  noted  that  to  achieve  the  required  Hausdorff  error, 
several  refinements  are  made  because  the  distance  from  the  compressed  surface  to  the  point  cloud 
is  large  due  to  the  fact  that  the  compressed  surface  on  a  given  cube  often  extends  far  away  from  the 
point  cloud.  To  resolve  this  difficulty  and  improve  encoding  the  idea  of  bounding  boxes  has  been 
introduced  and  analyzed.  Although  bounding  boxes  add  some  overhead  to  encoding,  this  overhead 
is  more  than  compensated  for  by  reducing  the  size  of  the  final  tree. 

When  receiving  the  compressed  file,  the  receiver  needs  an  algorithm  for  surface  reconstruction 
and  display  from  the  encoded  bitstream.  This  is  perhaps  the  most  significant  challenge  in  terrain 
encoding.  The  main  obstacle  is  the  identification  of  the  inside  and  outside  of  the  surface  from  the 
local  polynomial  fits.  In  other  application  domains  of  surface  compression,  this  is  accomplished  by 
determining  normals  to  the  surface.  These  are  either  assumed  to  be  given  by  the  sensor  or  else  are 
computed  using  the  high  resolution  of  the  point  cloud.  Neither  of  these  alternatives  are  available 
with  standard  terrain  point  clouds.  This  problem  has  been  resolved  during  the  grant  period  by 
determining  which  side  of  the  plane  is  in  the  interior  of  the  surface  using  sophisticated  labeling 
and  marching  algorithms.  These  algorithms  are  challenged  by  the  occlusions  and  missing  data. 
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Our  algorithm  resolves  this  by  assigning  cubes  of  high  altitude  as  out  and  those  of  low  altitude  as 
in.  Both  of  these  types  of  cubes  typically  have  no  data  in  them.  Then  a  marching  strategy  makes 
an  assignment  of  inside  and  outside  on  all  of  the  occupied  cubes.  This  is  done  so  that  there  is  a 
self  consistency.  The  portions  of  the  cube  are  then  unioned  to  obtain  a  solid  region  in  3D  whose 
boundary  is  the  surface.  Thus  the  surface  is  the  level  set  having  an  implicit  representation  of  the 
solid  region.  This  implicit  representation  is  important  because  it  allows  one  to  treat  features  such 
as  overhangs  and  wires  which  functional  representations  would  not  allow. 

The  main  advantage  of  using  an  implicit  representation  is  its  flexibility  and  robustness  in  deal¬ 
ing  with  complicated  topology  and  automatic  representation  of  holes  or  gaps  after  surface  recon¬ 
struction.  Implicit  representation  also  allows  one  to  generate  distance  fields  which  are  useful  for 
estimation  of  the  Hausdorff  metric,  autonomous  navigation,  line  of  sight  calculations  and  other  op¬ 
erations.  The  above  implicit  representation  is  typically  not  smooth.  However,  our  decoder  allows 
the  receiver  to  reconstruct  smoother  surfaces  using  the  technology  of  Mason- Petrova-Schaefer  [44] 
using  implicit  wavelet  fits. 

Octree  Representation  Under  the  Hausdorff  Distance:  We  previously  proposed  a  framework  for 
representing  a  point  cloud  with  a  parametric  union  of  piecewise  polynomial  surfaces  that  locally 
capture  the  geometry  of  the  point  cloud  at  various  scales  and  translations.  This  representation  is 
encoded  in  an  octree  data  structure,  where  children  nodes  encode  the  information  in  a  region  at  a 
finer  level  of  detail  than  what  is  captured  at  the  parent  node.  While  octree  representation  has  been 
well  studied  in  the  literature,  our  main  innovation  was  the  explicit  use  of  the  Hausdorff  distance  to 
control  the  fidelity  of  our  representation. 

The  Hausdorff  distance  measures  the  “worst-case”  deviation  between  the  point  cloud  and  sur¬ 
face.  In  contrast,  the  standard  least-squares  or  L2  metric  measures  only  average  deviation.  While 
the  L2  metric  is  standard  in  literature,  it  performs  rather  poorly  when  representing  thin  structures 
such  as  light  poles  or  telephone  wires  that  have  a  small  number  of  points.  Poor  representation 
of  these  thin  structures  can  have  a  negligible  impact  under  the  L 2  distance,  but  is  easily  detected 
under  the  Hausdorff  distance.  While  our  initial  strategy  used  the  Hausdorff  distance  to  measure  the 
fidelity  between  the  point  cloud  and  the  surface  representation,  the  underlying  fits  that  we  chose 
were  still  L2-optimal  fits.  The  reason  for  this  was  simplicity;  least-squares  fitting  is  well  under¬ 
stood,  easy  to  implement,  and  computationally  efficient.  By  contrast,  Hausdorff  optimal  fitting  is 
known  to  be  NP  hard.  While  our  previous  representation  scheme  is  functional,  there  is  often  a 
significant  gap  between  the  Hausdorff-optimal  and  L2-optimal  fit.  This  had  the  practical  effect  of 
requiring  an  excessive  number  of  nodes  to  represent  the  underlying  data. 

During  the  grant  period,  we  have  made  significant  progress  in  closing  this  performance  gap.  In 
particular,  we  have  devised  methods  for  creating  fits  that  are  optimal  in  the  d\  side  of  the  Hausdorff 
metric,  optimizing  the  distance  measured  from  the  point  cloud  to  the  surface.  The  one-sided  Haus¬ 
dorff  distance  from  the  points  to  the  plane  can  be  accomplished  easily  via  projection  and  taking  the 
maximum.  Since  the  one-sided  Hausdorff  distance  only  depends  on  the  distance  from  the  points 
to  the  surface,  it  can  be  computed  with  simple  projection  operations  and  taking  the  maximum. 
Because  of  this,  we  can  quickly  search  an  allowable  dictionary  of  fits  to  find  the  d\  -optimal  fit. 
Additionally,  we  have  shown  that  d\  -optimal  planar  fits  actually  provide  a  tight  bound  to  the  two- 
sided  Hausdorff  distance,  Dh .  This  bound  is  dependent  only  on  the  magnitude  of  the  one-sided 
Hausdorff  distance  and  the  size  of  the  interpoint  spacing  of  the  data.  The  interpoint  spacing  can 
be  solved  with  what  is  known  as  the  largest  empty  circle  problem  [47]  which  can  be  solved  in 
0{N\ogN)  time,  where  N  is  the  number  of  points  under  consideration.  This  need  only  be  solved 


once  per  node,  and  adds  little  computational  overhead  to  our  method.  This  bound  not  only  estab¬ 
lishes  the  worst-case  value  of  Dh  for  our  fit,  but  also  the  worst  case  Dh  for  any  planar  fit  available 
in  our  surface  dictionary. 

At  the  heart  of  our  encoding  scheme  is  the  issue  of  quantization.  We  wish  to  sufficiently  truncate 
the  cost  to  represent  a  surface  while  still  preserving  its  fidelity.  We  wish  to  avoid  using  an  excessive 
bitrate  for  encoding  while  simultaneously  providing  sufficient  resolution.  A  known  result  for  the 
L2  error  metric  is  that  one  must  double  the  resolution  of  coefficients  as  one  moves  from  high  order 
to  low  order  terms  [21]  to  achieve  optimal  encoding.  This  result,  as  one  might  expect,  does  not 
hold  under  the  Hausdorff  metric  for  the  case  of  planar  fits.  In  particular  we  have  shown  that  the 
Hausdorff  error  incurred  through  quantization  of  the  constant  term  is  identical  to  the  quantization 
in  the  linear  term.  This  allows  us  to  use  fewer  bits  in  the  constant  term  while  maintaining  the  same 
Hausdorff  fidelity,  which  reduces  our  overall  encoding  rate  significantly. 

We  demonstrated  the  performance  of  our  encoder  on  a  simple  data  set  involving  a  LiDAR  scan 
of  a  house.  We  compare  against  the  well  known  surflets  method  of  Chandrasekaran  [21],  which  is 
known  to  be  optimal  in  the  L 2  metric.  However,  because  their  method  employs  the  L2-distance  it 
has  difficulty  detecting  features  such  as  the  window  on  the  house  and  does  not  sufficiently  prune 
in  the  surface  fits  near  the  roof  until  the  tree  has  been  refined  to  an  extremely  deep  level. 


Figure  5.  Left:  Comparison  between  our  method  (Hausdorff  Tree)  with  the  L2 
Optimal  Method  of  Chandrasekaran,  et  al.  Our  method  achieves  superior  Hausdorff 
error  with  far  fewer  bits  in  the  final  encoding.  Center:  Hausdorff  tree  reconstruction 
at  1  bits/point.  Right:  The  surflets  reconstruction  at  1  bits/point  fails  to  capture  the 
window  of  the  house  and  does  not  adequately  prune  the  roof. 


2.  Learning-related  algorithms 

The  elements  of  Mathematical  Learning  theory  and  its  variants  are  particularly  relevant  to  syn¬ 
thesis  and  analysis  of  urban  terrain  data  in  several  respects.  USC,  Virginia  Tech,  and  TAMU  have 
a  long  ongoing  project  to  develop  novel  methods  to  optimally  learn  surfaces  from  point  cloud  data. 
For  example,  this  group  has  developed  new  theory  and  algorithms  for  sparse  occupancy  trees  [9], 
classification  from  sets  [5],  and  their  application  to  processing  point  cloud  data  [3]. 

As  described  in  Section  1.2,  the  implicit  use  of  its  principles  leads  to  algorithm  designs  for 
synthesis,  based  upon  local  statistics  and  metrics,  which  adaptively  partition  point  cloud  space  in 
order  to  construct  multi-level  approximate  surfaces  of  objects  imaged  by  sensors. 

In  Subsection  2.1,  we  describe  work  done  during  the  current  period  on  learning  surfaces  from 
point  clouds  where  metadata  is  available  on  sensor  position  and  pose.  In  this  case,  reconstruction 
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ambiguities  can  be  resolved  and  more  detailed  topological  properties  can  be  determined  in  sensor 
parameter  space.  See  Subsection  2.1  and  its  application  in  Section  5.  This  can  also  be  integrated 
with  registration  of  geometric  features  between  different  scans  in  a  SLAM  setting  in  order  to 
estimate  sensor  pose  and  position. 

Another  learning-related  problem  involves  properly  aligning  multiple  data  scans  of  a  targeted 
region  of  urban  terrain  with  the  objective  of  either  enhanced  imaging,  or  for  change  detection.  The 
data  may  be  collected  at  different  times  from  possibly  different  physical  sensor  states  or  by  using 
sensors  situated  at  multiple  locations.  For  proper  aggregation  of  the  data,  however,  reliable,  highly 
accurate  registration  of  the  collections  is  necessary,  otherwise  improper  registration  will  result  in  a 
significant  blurring  of  the  reconstruction  versus  that  which  is  produced  by  each  of  the  collections 
individually.  As  reflected  in  the  Eglin  push-broom  lidar  collection,  which  we  use  extensively  for 
testing,  multiple  passes  cannot  be  aligned  using  rigid  motion  registered  adjustments  alone.  This 
is  due  to  distortions  produced  by  noise  in  estimating  the  moving  sensor  platform’s  position  and 
pose,  but  also  significantly  by  the  fact  that  the  sensor  is  performing  line  scans  as  the  platform 
is  turning,  with  velocity  components  along  the  direction  of  the  turn.  If  the  sensor  line  scans  are 
left  to  right,  then  right  turns  result  in  elongation  in  the  scan  while  left  turns  lead  to  a  dilation.  In 
the  perceived  data,  this  has  the  effect  of  localized  nonlinear  distortions  (stretching/shrinking)  of 
an  imaged  surface.  In  Subsection  2.2  a  nonlinear  multi-level  point  cloud  registration  method  is 
proposed,  which  is  a  completely  analogous  extension  of  our  2D  work  on  time  series  scans  used 
in  microscopy  imaging  [4].  The  basic  localized  registration  procedure  is  described  and  applied  to 
multiple  airborne  lidar  swaths  to  demonstrate  the  algorithm’s  potential. 

Nonlinear  registration  involves  iteration  and  is  substantially  accelerated  by  identifying  a  set  of 
local  features  within  the  point  cloud  which  may  be  used  as  high  quality  markers  for  the  the  6 
degree  of  freedom  local  alignment.  Methods  for  extracting  structure  and  features  from  point  cloud 
data  are  reported  in  Subsection  2.3  and  Subsection  4.2,  as  reported  in  [33]  and  [16],  respectively. 
Further  as  the  data  sets  are  more  accurately  aligned  and  the  point  cloud  data  unioned,  more  accurate 
reconstructions  of  the  feature  points  themselves  may  be  obtained,  and  used  to  further  improve  the 
registration  process  applied  to  the  original  raw  point  clouds. 

Subsection  2.4  describes  joint  work  between  USC  and  TAMU  personnel  on  extracting  embed¬ 
ded  information  from  data  along  with  quantifiable  estimates  of  the  probability  and  risk  associated 
with  classification  of  the  information.  The  algorithms  developed  in  this  MURI  are  robust  with  re¬ 
spect  to  the  ambient  dimension.  For  example,  sparse  occupancy  trees  allows  processing  algorithms 
whose  complexity  is  proportional  to  the  size  of  the  data  set  rather  than  the  ambient  space  dimen¬ 
sion  thereby  defeating  the  curse  of  dimensionality  in  high  dimensional  settings.  This  is  important 
because  in  many  real  world  applications  of  point  cloud  data,  for  example  hyper  spectral  data,  can 
be  viewed  as  high  dimensional  problems.  Traditional  numerical  algorithms  suffer  from  the  curse  of 
dimensionality  which  make  them  useless  if  the  ambient  dimension  is  even  moderately  high  (larger 
than  10  for  example).  The  collaboration  between  USC,  VT,  and  TAMU  injects  new  models  for 
high  dimensional  functions  based  on  sparsity  and  variable  reduction  which  allows  processing  high 
dimensional  data.  For  example,  new  theory  and  algorithms  have  been  developed  for  classification 
in  [5]  by  exploiting  occupancy  trees  and  new  models  for  high  dimensional  functions.  These  algo¬ 
rithms  have  provable  high  order  convergence  under  margin  conditions  and  sparsity  assumptions. 
Query  algorithms  have  been  developed  in  [27]  and  [22]  which  break  the  curse  of  dimensionality 
through  variable  reduction  and  the  exploitation  of  state  of  the  art  results  from  perfect  hashing  and 
compressed  sensing. 
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2.1.  Adaptive  Partitioning  in  Sensor  Parameter  Space.  Highly  effective  surface  reconstruc¬ 
tions  from  point  clouds  generated  by  "push-broom"  line  scanners  have  been  previously  developed 
by  project  personnel  [34].  These  algorithms  have  been  based  upon  ideas  from  mathematical  learn¬ 
ing  and  approximation  using  adaptive  partitions  [8,  7,  11].  During  this  project,  USC  personnel 
have  implemented  the  analogue  of  these  algorithms  in  sensor  parameter  space  with  the  objective 
of  more  accurately  adapting  the  geometry  and  topology  of  sensed  structures  to  the  point  cloud. 
The  estimated  range  values  and  corresponding  adapted  geometry  from  the  sensor- state  registered 
parameter  space  are  then  mapped  unambiguously  to  Euclidean  world  coordinates.  This  may  be 
incorporated  into  a  larger  methodology  as  on-line  updates  to  the  3D  global  map,  or  efficiently 
time-coded  along  with  sensor  state  information  for  later  processing.  The  ranges  are  estimated 
over  progressively  adapted  cells  in  parameter  space,  going  from  coarse  to  fine  scales.  Cells  with 
no-returns  or  a  small  number  of  unorganized  points  are  then  deleted  and  marked  as  such.  Efforts 
have  focused  on  reconstructing  surfaces  from  sensed  point  clouds  in  which  ambiguities  between 
occluded  and  empty  regions  cannot  be  distinguished  due  to  missing,  but  necessary,  information  of 
the  sensor  lines  of  sight.  Sensor  frameworks,  such  as  those  being  designed  and  deployed  by  the  VT 
team  of  this  MURI,  typically  retain  time-coded  sensor  state  information  which  synchronizes  the 
data  streams  between  multiple  sensors  and  platforms.  This  added  metadata  enables  fast,  unambigu¬ 
ous  classification  of  occupied,  non-occupied,  and  occluded  regions,  especially  when  implemented 
in  an  adaptively  partitioned  computational  framework. 

The  natural  coordinate  system  for  many  classes  of  ranging  sensors  is  their  local  (angle, angle) 
parameter  space  in  which  spatial  information  provided  by  ranging  sensors  (e.g.  LIDAR,  optical, 
IR,  or  acoustic)  more  easily  settle  what  are  typically  ambiguous  spatial  questions  of  point  cloud 
data.  These  include  attributes  such  as  "occupied"  versus  "non-occupied"  surface  regions,  "interior" 
versus  "exterior"  designations  for  signed  distances  to  surfaces,  "occluded"  versus  "non-occluded" 
volumetric  regions,  and  "within  sensor  range  and  out  of  range"  regions,  to  name  several.  Once 
data  is  analyzed  using  this  available  metadata,  it  may  easily  be  transformed  to  update  the  surface 
models  in  global  coordinates  without  ambiguity  and  providing  guidance  to  fully  determine  any  un¬ 
sensed  regions  deemed  important  to  be  imaged  and  resolved.  Furthermore,  additional  properties 
such  as  intensity,  reflectance,  and  multiple  return  waveforms  can  be  encoded  for  classification  and 
feature  analysis,  as  described  in  Subsection  2.4  which  deals  with  learning  and  classification  in  high 
dimensions. 

Several  improvements  of  the  adaptive  partitioning  in  parameter  space  have  been  considered 
and  implemented.  Both  the  splitting/merging  criteria  and  the  local  data  fitting  have  been  based 
on  quantities  derived  from  the  representation  of  the  data  as  transformed  into  a  global  Cartesian 
coordinates  system,  but  using  the  implicitly  specified  line  of  sight.  The  main  goal  is  to  distin¬ 
guish  surface  segments  based  upon  range  estimates  which  appear  in  the  field  of  view  and  to  find  a 
good  approximation  to  each  surface  segment.  The  adaptive  triangular  mesh  defines  neighborhoods 
around  its  vertices  and  corresponding  local  point  clouds  representing  the  neighboring  data.  For  the 
local  fit  we  consider  not  only  fitting  by  a  single  plane  (e.g.  via  principal  component  analysis)  but 
also  analyzing  the  clustering  of  the  data  within  the  local  neighborhood  and  then  fitting  it  with  two 
(or  more)  different,  near-optimal  planar  segments.  The  latter  procedure  enables  the  possibility  of 
processing  several  different  surfaces  elements  with  relatively  sparse,  coarse  partitioning  and  better 
approximating  quality  than  traditional  dyadic  splitting.  In  addition,  we  analyze  appropriate  local 
basis  representations  using  singular  value  decomposition  to  process  one-dimensional  structures 
and  outliers.  A  PhD  student  in  Mathematics,  Kamala  Diefenthaler,  working  under  the  direction  of 
Peter  Binev  is  preparing  the  results  of  this  research  for  publication. 
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2.2.  Surface  registration  for  improved  reconstructions.  Operationally,  registration  of  lidar  data 
collections  is  necessary  for  many  tasks  involved  in  the  assimilation,  analysis,  interpretation,  and 
extraction  of  information  from  point  cloud  data.  For  example,  airborne  data  collection  typically 
requires  multiple  passes  over  a  target  region  in  order  to  provide  a  full  coverage  and  to  provide 
different  acquisition  views  to  compensate  for  missing  data  in  the  shadow  regions  due  to  occlusions. 
See  Figure  6(a)1,  where  color  corresponds  to  height.  Each  pass,  or  swath,  is  a  collection  of  sample 
lidar  points  which  should  be  properly  geolocated.  Due  to  inherent  errors  in  positioning  of  the 
sensor  platform  and  in  the  sensor  itself,  measurements  should  be  aligned  on  regions  of  overlap  with 
point  cloud  swaths  from  earlier  passes.  The  sensor  platform  movement,  along  with  its  positioning 
system,  introduces  complex  nonrigid  motions  and  therefore  induces  highly  nonlinear  deformations 
of  the  data  set.  These  add  significantly  to  the  positioning  errors  of  the  lidar  data  points.  In  the  case 
of  airborne  lidar  acquistion,  banking  maneuvers  of  the  aircraft  are  reflected,  e.g.,  in  the  swath 
patterns  in  Figures  6(b)  and  6(c). 

If  there  is  proper  correction  of  these  deformations  through  nonlinear  registration,  the  overlap 
regions  will  have  a  higher  density  of  sampled  points.  The  result  should  be  finer  resolution  of  the 
sampled  geometry  and  a  resulting  higher  fidelity  in  the  geometric  reconstructions.  Moreover,  reg¬ 
istered  multiple  passes  can  detect  slowly  moving  objects  within  sampled  environment,  or  provide 
enhanced  change  detection  capabilities.  In  Figure  6(a)  the  individual  swathes  (such  as  those  in 
Figures  6(b)  and  6(c))  were  painstakingly  post-processed  and  registered  by  hand  to  a  common  ref¬ 
erence  frame  over  a  period  of  months,  using  rigid  motions  to  compensate  for  errors  in  translation 
and  rotation.  Although  some  regions  were  well-aligned,  due  to  the  effects  of  non-rigid  motion  and 
nonlinear  distortions,  other  regions  were  not.  In  Figure  7,  we  have  overlaid  the  two  swaths  from 
the  Eglin  data  set,  rendering  one  in  green  and  the  second  in  red.  The  close-up  views  of  these  two 
data  sets  from  above  in  Figures  7(a)  and  from  the  side  in  7(b)  show  that  some  geometric  features 
are  not  aligned,  in  particular  the  light  pole  data  circled  in  blue.  This  misalignment  is  due  to  errors 
in  positioning  estimates  of  the  sensor  platform  by  the  aircraft’s  IMU.  If  the  data  points  from  the 
two  swaths  are  combined  and  processed  by  any  surface  approximation  method,  clearly  the  estima¬ 
tion  will  be  unsatisfactory,  most  likely  producing  two  sets  of  light  poles,  or  single  light  pole  with 
poorly  reconstructed  geometry  and  oversized  by  an  order  of  magnitude. 

In  order  to  address  the  nonlinear  distortions  of  positioning  in  point  cloud  collections,  we  describe 
the  current  effort  which  follows  our  recent  work  on  scanning  transmission  electron  micrographs 
to  formulate  the  registration  as  the  solution  to  a  nonlinear  minimization  problem  [4],  The  process 
developed  there  is  an  involved  iterative  procedure  operating  on  a  sequence  of  many  micrographs. 
We  describe  our  analogue  only  in  terms  of  only  two  swaths,  one  of  which  we  designate  as  the 
template  data  set  to  which  a  reference  data  set  is  to  be  registered.  Although  a  volume  registration 
analogue  can  likewise  be  used  for  signed  distances,  we  describe  only  the  surface  registration  ver¬ 
sion  for  unsigned  distances.  First  we  applied  geometric  reconstruction  algorithms,  developed  in 
this  MURI  program  (see  e.g.  Section  1.2),  to  each  of  the  swaths  to  obtain  relatively  good  quality 
surface  fits  Si,S2  C  M3  for  appropriate  subsets  within  the  overlap  region  of  the  swaths.  If  Si  is 
fixed  as  the  template  surface,  the  problem  then  is  to  determine  a  deformation  <|> :  R3  — >  R3  such 
that  the  surface  (j)(5i)is  ‘optimally  near’  the  surface  Si-  In  particular,  if  d$2  :  R3  — »  [0,°°)  denotes 
the  unsigned  distance  to  the  surface  Si,  our  idealized  matching  criterion,  i.e.  (|)(5i)  C  Si  can  then 
be  modeled  by  finding  the  <|)  from  an  appropriate  class  that  minimizes  the  functional 

^surface  [*}*]  =  /  (^(-*0 )  <75  (x) , 

JS\ 

Courtesy  of  Eglin  AFRL/MNG  VEAA  Data  Set  #1  by  permission 
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Figure  6.  (a)  Data  swathes  representing  multiple  passes  with  boxed  area  indicat¬ 
ing  one  region  of  overlap,  (b)  Swath  1  of  boxed  overlap,  (c)  Swath  2  of  boxed 
overlap.  Note  that  data  is  colored  by  height  above  bare  earth  for  visual  interpreta¬ 
tion. 

which  is  effectively  an  averaged,  one-sided  Hausdorff  distance  of  the  deformation  of  J>i  to  Si-  As 
a  first  order  registration,  we  model  <|>  using  a  parametrized  rigid  body  motion,  i.e.  <|>[oc,  (3,  y,  t]  (x)  = 
Rx(y)Ry(Wz(<*)x  + 1,  where  a,  (3  and  y  are  yaw,  pitch  and  roll  respectively,  and  t  £  R3  is  the 
translational  part  of  the  deformation.  The  registration  problem  at  this  coarse  level  is  solved  by 
minimizing  the  functional 

E [tt,  (3, y,  t]  =  ^surfacet^tot)  (3,  y,  t]] 

with  respect  to  the  deformation  parameters.  The  minimization  is  a  variant  of  that  proposed  in  [19] 
performing  an  explicit,  step  size  controlled  gradient  descent  where  the  corresponding  integrals  are 
evaluated  using  FEM  quadrature  approximations  for  the  finite  element  representation  of  J>i .  This 
basic  procedure  is  to  be  used  locally  at  each  level  from  coarse  to  fine  in  a  multi-level  framework 
to  build  the  nonlinear  deformation  map  <|>.  Figure  8(c)  illustrates  the  effectiveness  of  the  procedure 
when  applied  locally  to  the  template  and  reference  patches  in  Figures  8(a)  and  8(b). 

If  the  deformation  <|)  is  applied  to  the  entire  reference  swath,  the  case  for  nonlocal  registration  in 
a  multilevel  framework  becomes  clear  as  seen  by  the  mismatches  at  other  locations.  In  Figure  9(a) 
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(a)  (b) 


Figure  7.  (a)  Top  view  of  the  two  overlaid  swaths  with  lidar  point  cloud  data 
sensed  from  a  light  pole  circled  in  blue,  (b)  Close  up  side  view  of  the  two  point 
clouds  demonstrating  a  misalignment  of  the  two  point  clouds  around  the  light  pole. 


(a)  (b)  (c) 

Figure  8 .  (a)  local  reconstruction  of  light  pole  for  template  patch,  (b)  local  recon¬ 
struction  of  light  pole  for  reference  patch,  (c)  reference  patch  registered  to  template. 


where  the  overlaid  registered  reference  frame  is  viewed  from  the  side,  we  see  that  there  is  a  clear 
vertical  mismatch  with  substantial  separation  of  ‘bare  earth’  portions  between  the  two  swaths.  The 
background  light  poles  on  the  left  side  of  that  figure  and  the  building  edges  in  the  upper  right  of 
the  look-down  view  of  Figure  9(b)  show  strong  nonlinear  errors  in  horizontal  positioning. 

2.3.  Feature  Extraction  &  Mahalanobis  segmentation.  One  approach  of  better  analyzing  point 
cloud  structure  for  subsequent  surface  reconstruction  is  to  decompose  the  point  cloud  into  seg¬ 
ments  which  may  be  individually  modeled.  MURI  personnel  at  South  Carolina  have  developed 
an  orientation  invariant,  geometric  segmentation  algorithm,  SARG  (Self  Adjusting  Region  Grow¬ 
ing)  which  organizes  point  clouds  into  labeled  subclouds  whose  points  share  similar  features  [33] 
and  which  are  individually  are  more  amenable  to  reconstruction.  SARG  is  based  upon  the  clas¬ 
sical  Mahalanobis  statistical  method  for  comparison  of  two  normally  distributed  multivariate  dis¬ 
tributions  and,  implicitly,  to  distinguish  outliers.  The  distance  estimation  is  related  to  the  local 
Principal  Components  in  that  the  eigenvectors  of  the  covariance  matrix  are  used  as  a  coordinate 
system  whose  corresponding  eigenvalues  determine  the  scaling  factors  for  the  coordinate  direc¬ 
tions.  The  classical  Mahalanobis  distance  from  a  candidate  point  P  to  a  set  (i.e.,  labeled  segment) 
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(a)  (b) 


Figure  9.  (a)  Side  view  of  registered  reference  swath  to  template  swath,  (b)  Look 
down  view  of  same  overlaid  swaths  showing  significant  nonlinear  errors  in  building 
edges. 


S  =  {P\,Pi,  ■  ■  ■  ,Pn}  C  Rd  is  given  by 


(1)  dM(P,S)  :=  ^J(P-iu)TL-'  (P  —  fj). 

where  /./  is  the  center  of  mass  of  S  and  £  is  the  covariance  matrix  of  the  collection  S  —  /j.  To 
accommodate  the  important  cases  of  vanishing  eigenvalues,  this  definition  can  be  modified  by 
diagonalizing  the  covariance  matrix  as  £  =  Q  A  QJ  where  Q  is  the  matrix  with  eigenvectors  as 
columns  arranged  according  to  the  corresponding  (necessarily  nonnegative)  eigenvalues  A-i , . . . , 
The  distance  formula  in  (1)  may  then  be  generalized  and  reformulated  as 

<"w=-=a=o. 

°°  otherwise, 

where  jo  is  the  number  of  non-zero  eigenvalues  of  £  counted  with  their  multiplicities,  and  P  := 
(Pi,...  ,Pd)T  ■=  QT(P  —  ju)  are  the  coordinates  of  P  in  the  new  coordinate  system.  The  SARG 
algorithm  iteratively  grows  the  current  segment  capturing  all  points  in  the  cloud  which  are  judged 
to  be  close  in  this  distance.  The  algorithm  may  be  implemented  with  computational  complexity 
0(NlogN)  and  has  proved  effective  for  several  fielded  scanning,  flash,  and  burst  illumination  lidar 
systems  See,  for  example,  Figures  10-11  below.  Similar  results  were  obtained  for  data  obtained  by 
the  Virginia  Tech  dual  axis  lidar  system  and  were  presented  during  the  Year  2  review. 

In  Figures  10  a)-d),  the  SARG  algorithm  is  applied  to  scanning  LIDAR  data  provided  by  NATO 
SET11 8  collaborators  from  the  Defense  Research  and  Development  Canada  -  Valcartier  for  se¬ 
lected  testing  of  MURI  algoritms.  Image  a)  depicts  the  original  LIDAR  point  cloud.  Image  b)  is 
a  rendering  of  the  SARG-segmented  subclouds  labeled  by  various  colors,  with  Person  2  colored 
as  magenta.  A  multiresolution  surface  reconstruction  is  shown  for  these  two  subclouds  in  Fig¬ 
ures  10  c)-d),  respectively.  Higher  quality  reconstructions  are  provided  in  Figure  17  using  UC-I’s 
three  dimensional  binary  image  segmentation  formulation  described  in  Section  4.2. 


(2) 


<Im(P,S)  := 
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C)  d) 


FIGURE  10.  Iterated  Mahalanobis  segmentation  of  scanning  LIDAR  data  provided 
by  DRDC-Valcartier  NATO  SET118  collaboration,  a)  LIDAR  Point  Cloud;  b)  seg¬ 
mented  subclouds  labeled  by  color;  c)  surface  reconstruction  for  person  1 ;  d)  surface 
reconstruction  for  person  2. 

Figures  1 1  a)-f)  presents  the  results  of  SARG  applied  to  burst  illumination  LIDAR  data  provided 
by  NATO  SET118  collaborators  from  the  Defence  Science  and  Technology  Laboratory  (UK  Min¬ 
istry  of  Defence).  Figure  11  a)  is  a  photo  of  the  imaged  scene,  with  parts  b)-c)  representing  side 
and  front  views  of  the  sign  in  the  foreground.  In  these  images  magenta  used  to  color  the  residual 
set  after  the  iterated  segmentation.  Figure  1 1  d)  is  a  rendering  of  the  surfaces  constructed  from  the 
segmented  sub-clouds  along  with  the  original  data.  Figures  1 1  e)-f)  show  similar  views  from  the 
side  and  front  for  the  vehicle  in  the  background  of  the  photo  in  part  a). 

The  SARG  algorithm  is  naturally  suited  for  implementation  in  CUDA  for  GPU  (Graphical  Pro¬ 
cessing  Units)  which  will  compensate  for  its  more  expensive  0(Alog N)  performance  to  enable  its 
potential  for  real-time  processing  on  small  platforms.  Other  known  bottlenecks,  although  still  lin¬ 
ear  in  computational  complexity,  have  previously  been  implemented  in  CUDA,  including  multires¬ 
olution  PCA,  complex  wavelets,  and  the  chamfer  algorithm  for  estimating  the  Hausdorff  metric. 
Speedups  of  forty  or  more  have  been  achieved  for  these  algorithms  on  standard  graphics  proces¬ 
sors,  but  requires  a  careful  study  of  algorithm  design  for  memory  management. 

2.4.  High  Dimensional  Learning  and  Classification.  In  many  application  domains  of  mathe¬ 
matical  learning,  point  cloud  data  can  be  viewed  as  random  independent  draws  of  some  unknown 
probability  distribution  p  and  problems  are  formulated  as  ones  of  either  regression  or  classification. 
Classification  problems  are  key  to  resolving  several  practical  issues  that  emerge  in  processing  point 
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e)  f) 


FIGURE  1 1 .  Iterated  Mahalanobis  segmentation  applied  to  3D  burst  illumination 
LIDAR  data  provided  by  Dstl-UK  NATO  SET118  collaboration,  a)  Photo  of  test 
scene  with  target  sign  in  foreground  and  vehicle  in  background;  b)  side  view  of 
segmented  subclouds  around  target  sign  with  ‘noise  label’  magenta  ;  c)  front  view 
of  segmented  subclouds  around  target  sign;  d)  reconstructed  surfaces  around  tar¬ 
get  along  with  point  cloud;  e)  side  view  of  background  vehicle;  f)  front  view  of 
background  vehicle. 


clouds  and,  in  particular,  problems  on  extracting  embedded  information  from  urban  terrain  data. 
The  development  of  a  theory  to  provide  quantifiable  estimates  of  the  probability  of  classification, 
based  upon  minimization  of  a  risk  functional,  is  therefore  extremely  important  for  decision  makers 
and  in  the  design  of  reliable  classification  algorithms. 
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We  are  developing  new  theory  and  algorithms  for  classification  directed  at  high  dimensional 
problems,  such  as 

•  the  analysis  of  chemical-biological  gas  plume  concentration  levels  moving  through 
an  urban  environment  imaged  by  frequency  agile  lidar  (FAR)  measurements.  Here 
by  processing  a  set  of  observations,  e.g.  spatial/temporal  multichannel  spectral 
measurements,  the  goal  of  classification  of  the  chem-bio  agents  is  aimed  so  that  false 
alarms  of  threats  are  reduced  while  incurring  very  low  risk. 

•  classification  of  conventional  lidar  point  clouds  by  signatures  (based  on  geometry, 
topology,  reflectance,  distribution  of  multiple  returns)  into  sets  designated  as  either 
man-made  structures  (including  buildings,  roadways,  bridges,  power  lines,  walls,  or 
vehicles)  or  natural  features  (such  as  waterways  or  vegetation).  The  classification  geo¬ 
metric  "signatures"  may  be  used  for  relational  database  queries  to  determine  possible 
candidate  locations  within  a  region  for  a  designated  object  of  interest. 

In  these  example  cases,  each  of  the  observations  detects  several  characteristics  of  objects.  The 
vector  of  these  characteristics  can  be  considered  as  a  point  in  a  high  dimensional  space  X.  Based 
on  some  additional  knowledge,  the  objects  and  their  corresponding  points  from  X  are  grouped  in 
classes.  The  classification  problem  processes  the  preliminary  (training)  observations  in  such  a  way 
that  for  every  new  query  from  X  we  are  able  to  predict  its  class  with  preassigned  high  confidence. 
Typically,  the  outcome  is  considered  +1  if  the  point  is  from  a  designated  class  of  interest  and 
—  1  otherwise.  Ideally,  we  can  assume  that  the  observations  are  independent  drawings  from  a 
probability  measure  p  over  the  cross  product  of  the  space  X  with  Y  :=  {+1,  —  1}.  If  for  a  given 
point  x  from  X  we  define  T|(x)  =  E(y  |x)  as  the  expectation  of  y  given  x,  then  the  best  classifier, 
called  the  Bayes  classifier,  is  given  by  the  set  Q*  of  all  points  x  from  X  for  which  the  expected 
value  T)  (x)  of  the  outcome  is  positive.  By  assigning  + 1  to  the  points  from  Q*  and  —  1  for  the  ones 
outside,  the  measure  R(Q*)  of  the  misclassified  points,  called  the  risk,  is  minimized. 

The  goal  of  the  empirical  classification  algorithms  is  to  find  a  good  approximation  Q  to  the  set 
Q*  using  a  limited  variety  of  approximating  sets.  Given  a  family  F  of  subsets  of  X,  we  consider  the 
square  of  the  expected  value  of  the  outcomes  for  a  set  S  from  this  family.  If  this  quantity  is  larger 
than  t  times  the  measure  of  S,  the  set  S  is  called  t-reliable.  For  sufficiently  large  t,  this  property 
guaranties  that  the  prediction  based  on  the  empirical  estimates  for  the  sign  of  the  outcomes  on  the 
set  S  will  be  correct  with  high  probability. 

Our  theoretical  analysis  involves  the  smoothness  properties  of  the  regression  function  r|  around 
the  boundary  of  Q*.  This  allows  us  to  construct  an  adaptive  procedure  for  building  the  family  F  of 
approximating  sets  that  does  not  use  explicitly  knowledge  about  this  smoothness.  We  denote  by  Tp 
the  integral  of  r|  over  a  subset  S  of  X  with  respect  to  the  measure  px  induced  by  p  on  X.  By  ps  we 
denote  the  amount  of  this  measure  on  the  set  S.  Then,  the  ^-reliability  is  expressed  by  |rp|2  >  t  p^. 
If  the  sets  S  are  the  building  blocks  used  to  assemble  the  family  F,  then  it  is  important  to  limit  the 
influence  of  the  sets  that  are  not  t-reliable.  One  particular  line  of  investigation  is  to  find  a  setup  in 
which  the  total  measure  of  the  unreliable  sets  can  be  reduced.  In  the  case  that  we  have  a  complete 
knowledge  of  p,  this  could  be  done  easily  by  building  a  family  F  that  resolves  well  the  boundary 
of  Q* .  In  general,  we  do  not  have  any  information  about  p  and  one  can  build  F  based  on  the 
assumption  that  p  has  certain  properties.  An  example  for  such  an  assumption  is  the  Tsybakov  noise 
condition  that  requires  that  the  measure  of  the  set  of  all  x  from  X,  for  which  |t|(x)  |  <  8  is  limited 
by  a  constant  times  the  q-th  power  of  8.  Many  classification  algorithms  are  theoretically  validated 
based  on  estimates  assuming  this  condition.  However,  it  is  designed  primarily  for  reducing  the 
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total  risk  R(Q )  of  the  approximating  set  Q.  Since  this  risk  cannot  be  below  R(Q*),  the  actual  goal 
should  be  to  find  conditions  under  which  the  excess  risk  R(Q)  —  R(Q*)  would  be  reduced.  This 
motivates  our  investigations  to  design  a  different  theoretical  setup  aiming  at  incorporating  such 
conditions  and  by  this  to  enlarge  the  set  of  measures  p  for  which  it  applies.  Our  main  tool  is  a 
specific  modulus  co(p,e)  that  analyzes  the  variance  term  for  a  given  classification  algorithm.  The 
threshold  parameter  £  =  e(S)  is  designed  to  estimate  the  error  of  the  empirical  approximation  of 
T|s.  It  can  be  adjusted  to  provide  estimates  based  on  Tsybakov  noise  condition  but  other  choices 
give  more  general  results,  in  particular  for  r|(jc)  belonging  to  smoothness  classes  similar  to  ones 
defined  by  Besov  spaces. 

Motivated  by  the  above  mentioned  theoretical  findings  we  design  partition-based  empirical  al¬ 
gorithms  for  classification.  Some  of  the  advantages  of  our  procedure  include  its  online  data  assim¬ 
ilating  capabilities  and  fast  computation. 

We  build  a  full  binary  tree  corresponding  to  adaptive  partition  rules  using  the  paradigm  of  sparse 
occupancy  trees  [10]  to  handle  the  data.  Given  a  specific  t,  we  then  find  through  a  thinning  process 
the  finest  possible  partition  P  such  that  for  each  leaf  node  in  the  corresponding  tree,  either  it  or 
its  sibling  is  part  of  a  t -reliable  set  S.  The  set  S  can  be  a  cell  of  the  current  partition  or  union  of 
several  adjacent  cells  from  it.  In  the  case  of  simplicial  partitions  we  can  choose  the  sets  S  to  be 
the  simplices  from  the  current  partition  adjacent  to  a  vertex.  Once  the  partition  P  is  found,  we 
calculate  empirical  estimates  for  the  local  excess  risk  at  each  node  of  its  tree  going  fine-to-course 
and  assuming  that  the  excess  risk  is  0  at  the  leaf  nodes.  Based  on  these  estimates  we  then  find 
a  near-best  tree  approximation  P*  of  the  partition  for  Q*  with  m  =  m(t )  elements.  Since  m  is  an 
increasing  function  of  t,  we  can  adjust  t,  based  on  the  comparison  of  the  trees  for  P  and  P* ,  to  avoid 
underfitting  or  overfitting.  The  set  Q  is  then  defined  as  a  union  of  all  cells  S  of  P*  with  rp’  >  0.  This 
algorithm  is  universal  in  the  sense  that  it  does  not  assume  any  prior  knowledge  of  the  smoothness 
of  T|(x).  In  addition,  one  can  be  more  conservative  in  forming  the  set  Q  and  only  include  cells  S 
with  r\s  >  0  that  are  part  of  reliable  sets  reducing  the  potential  of  false  positives. 

This  new  theory  and  algorithms  which  exploits  occupancy  trees  and  new  models  for  high  di¬ 
mensional  functions  has  been  developed  for  classification  in  [5],  These  algorithms  have  provable 
high  order  convergence  under  margin  conditions  and  sparsity  assumptions. 

Query  algorithms  have  also  been  developed  in  [27]  and  [22]  which  break  the  curse  of  dimension¬ 
ality  through  variable-reduction  and  the  exploitation  of  state  of  the  art  results  from  perfect  hashing 
and  compressed  sensing. 


3.  Estimation  and  Approximation  from  Sensor  Networks 

As  imaging  sensors  have  become  more  compact  over  the  past  decade,  there  has  been  increasing 
interest  in  deriving  estimation  and  approximation  specifically  for  classes  of  sensor  networks.  Dur¬ 
ing  the  last  year,  the  MURI  team  has  continued  to  investigate  how  the  approximation  and  learning 
theoretic  approaches  developed  by  the  team  can  be  extended  to  this  important  class  of  problems. 
Significant  progress  has  been  made  through  the  collaboration  of  Princeton  University,  Virginia 
Tech,  the  University  of  South  Carolina  and  Texas  A&M  University. 

Over  the  past  year,  the  efforts  at  Princeton  in  support  of  this  MURI  have  been  in  the  broad  area  of 
distributed  learning  and  pattern  recognition.  The  Princeton  team  of  the  MURI  has  focused  on  the 
following  two  problems:  (1)  semi-supervised  consensus  segmentation  and  (2)  distributed  sensor 
localization. 
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3.1.  Graphical  Models  for  Distributed  Learning.  Reproducing  kernel  methods  are  a  popular 
and  highly  successful  approach  to  nonparametric  learning  and  pattern  recognition.  These  methods 
takes  as  input  a  training  set  U ™=1S,-  =  S  =  {(XJ-,y,)}^=1  and  in  the  least-squares  regression  setting 
output  a  function  fx  :  X  — >  y  which  solves  the  optimization  problem 


(3) 


min 


i=  1 


Here,  Xk  is  a  reproducing  kernel  Hilbert  space  to  which  the  regression  function  is  constrained 
to  belong.  This  space  is  induced  by  a  kernel  function  K and  the  minimizer  f\  E  Xk  of  the 
optimization  problem  can  be  written  in  the  form 


m 

(4)  fx(-)  =  Y,cxK(;Xi) 

1=1 

Solving  the  optimization  problem  is  difficult  in  distributed  settings  with  resource  constraints, 
when  there  are  a  collection  of  sensors,  each  observing  an  example  ( Xi,Yi )•  We  have  explored 
methods  for  solving  this  problem  that  relies  on  local  (and  iterative)  sharing  of  data,  not  entire 
functions,  and  thereby  addresses  the  practical  weakness  that  sometimes  limits  the  applicability  of 
other  methods  (e.g.,  the  incremental  subgradient  approach). 

In  this  approach,  each  sensor  i  queries  its  neighbors’  data  and  uses  this  local  data  to  compute  a 
global  estimate  for  the  field  by  solving 


(5) 


min  'S' 

fertK  '-jeNi 


(f(xj)-yj)2+ 


We  can  iterate  through  the  network  allowing  each  sensor  to  compute  a  global  estimate  using 
only  local  data.  The  key  idea  for  propagating  information  is  to  couple  this  iterative  process  using  a 
set  of  message  variables.  Multiple  passes  through  the  network  are  made.  The  resulting  algorithm 
is  depicted  pictorially  in  the  figure  below. 
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Figure  12.  Training  Distributively  with  Alternating  Projections 

This  approach  is  inspired  by  methods  from  graphical  models,  belief  propagation,  message¬ 
passing,  etc.  The  approach  is  independent  of  assumptions  that  couple  the  kernel  with  the 

network  topology.  Thus,  prior  domain  knowledge  about  P xy  (the  joint  distribution  of  (X.Y))  can 
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be  encoded  in  the  kernel;  the  training  algorithm  approximates  the  centralized  estimator  as  closely 
as  the  communication  constraints  allow.  Secondly,  sensors  only  share  estimates  of  functions  at  a 
small  number  of  points,  and  not  the  functions  themselves.  This  significantly  broadens  the  scope 
of  problems  where  the  approach  is  applicable.  During  this  MURI,  we  made  contributions  to  this 
approach  to  distributed  learning  and  pattern  recognition  in  a  variety  of  ways. 

3.2.  Semi-supervised  Consensus  Clustering  and  Segmentation.  We  developed  a  new  method 
for  learning  how  to  achieve  consensus  among  several  segmentation  outputs.  The  suggested  method 
employs  prior  information  about  the  target  objects  to  be  detected  after  the  segmentation  step  in 
order  to  learn  the  distance  metrics  and  the  weight  parameters  of  classical  consensus  clustering 
algorithms. 

Initially,  several  segmentation  algorithms  are  used  independently  with  different  parameter  sets 
as  base  segmentation  algorithms  to  segment  the  remote  sensing  images.  Then,  the  segmented 
images  are  fused  in  order  to  reach  a  consensus  on  the  sets.  The  proposed  method,  called  Semisu- 
pervised  Consensus  Clustering,  is  based  on  the  classical  Filtered  Stochastic  Best  One-Element- 
Move  (BOEM)  algorithm.  This  method  is  reformulated  to  incorporate  the  prior  information  to 
semi- supervise  the  consensus  clustering  process.  For  this  purpose,  we  employ  must-link  and  must- 
not-link  constraints  in  the  optimization  problem. 

We,  also,  define  and  compute  the  partition  weights  in  Filtered  Stochastic  BOEM.  Finally,  we 
suggest  a  learning  method  to  enhance  the  distance  function  and  algorithm  parameters.  We  have 
applied  our  new  semisupervised  consensus  segmentation  method  to  the  segmentation  of  hyper- 
spectral  and  aerial  remote  sensing  images.  We  have  studied  the  consensus  on  the  segmentation 
outputs  of  k-Means,  Normalized  Cuts,  Graph  Cuts  and  Meanshift  algorithms. 

3.3.  Aggregating  Probabilistic  Forecasts.  We  have  continued  our  efforts  to  develop  methods 
for  combining  forecasts  of  probability  of  complex,  interrelated  events  from  multiple  forecasters. 
Low  level  algorithms  can  be  utilized  to  provide  probability  forecasts  for  a  broad  range  of  events. 
Our  techniques  provide  a  means  for  fusing  this  information  for  higher  level  processing.  With  a 
large  number  of  sensors  and  even  larger  number  of  events,  one  of  the  main  challenges  is  to  find 
computationally  feasible  algorithms.  We  have  developed  methods  based  on  alternating  projections 
that  provide  excellent  results  in  reasonable  time. 

As  part  of  this  MURI,  we  developed  a  new  algorithm  for  combining  probabilistic  assessments 
from  a  large  pool  of  judges,  with  the  goal  of  efficiently  implementing  the  Coherent  Approximation 
Principle  (CAP)  while  weighting  judges  by  their  credibility.  We  introduced  two  measures  of  a 
judge’s  likely  credibility,  and  we  used  these  to  determine  the  judge’s  weight  in  aggregation.  The 
algorithm  was  applied  to  a  dataset  of  nearly  half  a  million  probability  estimates  of  events  related 
to  the  2008  US  presidential  election  (over  16000  judges).  Compared  with  simple  scalable  CAP 
algorithms,  our  schemes  improved  the  stochastic  accuracy  with  a  comparable  run  time,  demon¬ 
strating  the  effectiveness  of  the  weighting  methods  for  aggregating  large  numbers  and  varieties  of 
forecasts.  Also,  we  found  that  smart  weighting  allows  particular  subsets  of  events  to  be  aggregated 
more  accurately  and  comparison  with  probabilistic  predictions  from  popular  prediction  markets 
and  poll  aggregators  showed  excellent  performance  of  the  weighted  CAP  algorithms. 

We  investigated  two  further  methods  to  improve  the  forecasting  accuracy  within  the  CAP  frame¬ 
work  and  developed  practical  algorithms  for  these.  These  methods  allow  flexibility  to  add  fixed 
constraints  to  the  coherentization  process  and  to  compensate  for  the  psychological  bias  present  in 
probability  estimates  from  human  judges.  The  algorithms  were  tested  on  the  election  data  set  and 
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the  results  showed  that  both  methods  improve  the  stochastic  accuracy  of  the  aggregated  forecasts 
compared  to  using  simple  CAP. 

3.4.  Geometric  Estimation.  We  investigated  several  problems  in  geometric  estimation.  That 
is,  given  an  underlying  geometric  object  (shape,  surface,  trajectory,  etc.),  we  are  interested  in 
estimating  the  object  given  different  forms  of  noisy  measurements.  In  particular,  we  studied 
shape/surface/trajectory  reconstruction  based  on  point  cloud  data  (potentially  with  time  stamps) 
in  a  noisy  environment. 

We  developed  a  noise  resistant  ellipse/spheroid  fitting  algorithm  with  an  innovative  objective 
function.  Our  method  provides  more  accurate  axial  direction  estimation  in  noisy  environments. 
We  combined  this  new  objective  function  with  an  efficient  iterative  algorithm  with  a  correction 
term  so  that  it  can  obtain  accurate  axial  estimation  as  well  as  accurate  fitting  of  the  size  of  the 
ellipse/spheroid. 

To  better  deal  with  outliers  in  ellipse  fitting,  and  more  generally,  in  curve  and  surface  fitting,  we 
developed  a  hybrid  outlier  detection  algorithm  combining  both  proximity-based  and  model-based 
outlier  detection  techniques.  Our  hybrid  technique  can  effectively  eliminate  outliers  of  various 
types,  and  considerably  improves  the  robustness  of  ellipse/spheroid  fitting  for  scenarios  with  a 
large  proportions  of  outliers  and  high  levels  of  inlier  noise. 

A  third  problem  we  considered  in  this  area  is  joint  to  shape  and  trajectory  reconstruction  of 
rigid  bodies  from  distributively  collected,  asynchronous  point  cloud  data  with  time  stamps.  We 
developed  an  energy-minimization  scheme  is  to  solve  the  trajectory  reconstruction  problem  of  rigid 
bodies  with  known  shape  parameters,  assuming  that  the  rigid  body  moves  in  an  energy  efficient 
manner  with  an  acceleration  upper  limit.  We  generalized  this  basic  method  is  to  the  case  with 
unknown  rigid  body  shape  parameters,  employing  cross-validation  techniques  to  determine  the 
best  parameter  hypothesis. 

3.5.  Distributed  Sensor  Localization.  Localization  is  an  essential  tool  in  many  sensor  network 
applications.  Over  the  years,  a  rich  literature  has  been  developed  to  solve  the  sensor  localization 
problem  from  different  perspectives.  Four  criteria  are  anchor-based  vs.  anchor  free,  incremental 
vs.  concurrent,  fine-grained  vs.  coarse-grained,  and  centralized  vs.  distributed.  The  spring  model 
algorithm  we  have  developed  is  a  distributed,  concurrent,  fine-grained  algorithm  to  solve  the  sensor 
localization  problems.  We  have  focused  on  an  anchor-based  scenario  so  as  to  obtain  absolute 
coordinates  of  the  sensors  in  the  network.  Our  central  ideas  are  composed  of  three  techniques  to 
improve  the  spring  model  algorithm  for  solving  localization  problems  in  a  wireless  sensor  network 
with  anchors. 

First,  we  solve  the  two-dimensional  localization  problem  in  a  three-dimensional  space.  This 
dimension  expansion  technique  can  effectively  prevent  the  spring  model  algorithm  from  falling 
into  local  minima. 

Second,  the  Hooke  spring  force  (linear  force,  or  quadratic  potential)  is  generalized  to  spring  with 
an  Lp  potential  function.  The  optimality  of  different  values  of  p  is  compared  under  different  noise 
environments.  Our  simulation  results  show  that  p  =  2,  i.e.  Hooke  force,  is  good  choice  of  spring 
for  Gaussian  noise,  yet  for  Laplacian  noise,  p  =  1.3  outperforms  other  choices  of  Lp  potential 
functions. 

Third,  we  introduce  a  customized  spring  force  function  with  "lock-in"  mode,  which  has  larger 
strength  when  the  estimated  distance  between  two  sensors  is  close  to  the  true  length  of  the  spring, 
is  introduced,  which  accelerates  the  convergence  speed  of  the  algorithm. 
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All  these  techniques  significantly  improve  the  noise  resistance  and  efficiency  of  the  spring  model 
algorithm.  We  demonstrated  the  efficacy  of  these  algorithms  by  multiple  simulations,  especially  in 
a  scenario  with  anchor  points  of  longer  broadcasting  radius  than  other  sensors. 

3.6.  Consensus  and  Approximation  Theory.  Building  on  the  historical  contributions  of  Prince¬ 
ton  in  decentralized  optimization,  researchers  at  Virginia  Tech  have  concentrated  during  the  last 
year  on  extending  their  results  using  the  approximation  theoretic  results  obtained  by  the  University 
of  South  Carolina  and  Texas  A&M  University. 

3.6.1.  The  Global  Optimization  Problem.  We  consider  a  network  of  n  vehicles  labeled  i  =  1 . . .  n 
whose  communication  connectivity  is  described  by  a  graph  §  :=  {‘U,  £}  where  V  denotes  the 
nodes  and  £  defines  the  edges  in  the  graph.  Individual  vehicle  nodes  may  be  airborne,  marine 
or  terrestrial.  We  suppose  each  vehicle  i  collects  local  measurements  z'  =  {(x^ylk)} kez+  over 
time  steps  k  =  1,2, ... ,  that  represent  some  problem  specific  external  field  /  of  interest.  Several 
assumptions  on  the  measurement  processes  and  network  communication  motivate  the  research  in 
this  proposal.  We  assume  that  (Al)  the  measurement  processes  have  extremely  high  bandwidth 
measured  in  bits  per  second,  (A2)  the  communication  bandwidth  over  the  channels  described  by 
the  graph  Q  is  insufficient  to  share  the  measurement  data  in  real  time,  (A3)  the  computing  and  stor¬ 
age  capacity  of  each  node  is  limited,  and  (A4)  the  desired  behavior  of  the  network  or  collective  can 
be  expressed  in  terms  of  some  globally  defined  cost  or  loss  function  J7.  The  loss  or  performance 
function  J  depends  on  the  observations  by  all  agents  z  '■=  {zl , Z2 , . . . ,  z"  } ,  or  is  approximated  by  an 
empirical  loss  function  J7Z  that  depends  on  the  observations.  In  either  case,  we  suppress  the  depen¬ 
dency  on  the  observations  z  in  the  discussion  that  follows.  The  data  that  is  collected  by  the  network 
is  to  be  used  to  find  the  solution  of  the  infinite  dimensional,  distributed  optimization  problem  where 
we  seek  the  collection  of  estimates  /*  :=  {f*1, . .  .,/*”}  e  H  that  solves  the  optimization  problem 

/*=  argmin  J(f) 

where  H  :=  d{1  x  •  •  •  x  d{n  is  the  hypothesis  space  of  admissible  agent  solutions,  J7  is  the  loss 
or  cost  function  to  be  minimized  and  B(/)  encodes  the  constraints  imposed  among  the  solutions 
achieved  by  the  agents.  The  constraint  operator,  for  example,  can  impose  conditions  that  enforce 
communication  compatibility  as  dictated  by  the  connectivity  graph  (j.  Numerous  applications 
of  interest  to  the  Army  including  distributed  exploration  and  mapping,  mine  detection,  hazard 
detection,  distributed  path  planning  and  multivehicle  control  can  be  expressed  in  the  form  above 
for  distributed  vehicle  teams. 

3.6.2.  Local  Problems  and  the  Learning  Dynamic.  Our  general  approach  to  the  solution  of  the 
above  global  optimization  problem  uses  the  vehicle  team  to  construct  a  distributed  computing  sys¬ 
tem  that  generates  successive  approximations  that  converge  to  the  global  solution.  In  our  strategy 
the  network  of  agents  construct  a  sequence  of  approximations  of  the  original  optimization  problem 
via  a  two  stage  learning  dynamic.  In  this  dynamic,  the  agents  use  their  most  recent  data  to  solve 
local  optimization  problems  and  subsequently  update  these  local  solutions  using  functionals  that 
depend  on  the  shared  data.  At  time  step  e  M+  during  the  local  optimization  stage,  each  agent  i 
constructs  a  local  optimal  estimate  f[,  that  is  the  solution  to 

fk  ■=  argmin  fk(f) 
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where  Jlkf)  is  a  local  cost  function.  Once  the  local  optimal  estimates  fk  :=  have 

been  generated,  shared  information  functionals  pk  :=  p^Cfk)  are  constructed  which  define  infor¬ 
mation  that  is  communicated  over  the  network.  Finally,  during  the  communication  sharing  and 
update  stage,  each  agent  constructs  its  final  estimate  at  time  step  t *  £  R+  using  its  current  local 
estimate  flk  and  the  shared  information  functionals 

where  T^(-)  is  a  nonlinear  update  operator. 

The  two-stage  learning  dynamic  above,  which  is  described  in  more  detail  in  [23,  24],  is  quite 
general  in  structure.  Several  questions  are  addressed  in  our  research.  The  first  question  regards  the 
convergence  of  the  final  collection  of  estimates  to  the  solution  of  the  global  optimization  problem 

(6)  ft  :=  {ft, ....  ft  }->/*  :={/M, 

Many  different  authors  have  studied  sufficient  conditions  for  convergence  in  Equation  6  in  the  spe¬ 
cial  case  that  fk  evolves  in  some  finite  dimensional  space.  The  focus  of  our  study  is  the  selection  of 
appropriate,  infinite  dimensional  approximation  spaces  for  evolution  of  the  sequence  of  estimates 
{fk}ken-  This  viewpoint  has  two  important  implications.  The  first  is  that  synthesis  of  techniques 
from  approximation  theory  as  outlined  in  [8,  25]  enables  the  derivation  of  rates  of  convergence 
that  are  optimal  up  to  a  logarithmic  term.  These  estimates  have  no  counterpart  in  methods  that 
assume  a  priori  that  estimates  evolve  in  a  finite  dimensional  space.  The  second  implication  is  that 
powerful  methods  for  compression,  encoding  and  decoding  of  shared  information  are  obtained. 
These  methods  provide  a  rigorous  and  systematic  method  for  modulating  the  bandwidth  of  com¬ 
munication  across  the  network.  This  can  be  a  critical  feature  in  applications  to  autonomous  vehicle 
networks  where  communication  bandwidth  is  extremely  limited. 

3.6.3.  Progress  in  Consensus  Estimation  in  Infinite  dimensional  Spaces.  The  general  strategy 
above  has  been  successfully  developed  by  the  authors  with  extremely  promising  preliminary  re¬ 
sults.  By  developing  the  framework  for  consensus  approximation  as  a  distribution  free  learning 
problem,  the  solution  of  the  problem  is  achieved  via  a  distributed  optimization  of  a  collective  error 
function  subject  to  consensus  constraints.  This  approach  is  novel  and  noteworthy  in  the  following 
respects: 

(1)  The  consensus  estimation  problem  is  posed  as  a  distributed  optimization  problem  among 
agents  in  (infinite  dimensional)  approximation  spaces. 

(2)  The  decentralized  optimization  problem  is  construed  as  a  stochastic  optimization  problem 
where  the  probability  distributions  that  characterize  agent  sampling,  as  well  as  the  smooth¬ 
ness  or  approximation  properties  of  the  unknown  function,  are  not  known  a  priori. 

(3)  The  theoretical  framework  naturally  lends  itself  to  compression,  encoding  and  decoding 
of  the  shared  information  functionals.  The  parsimonious  exchange  of  information  among 
nodes  can  be  critical  to  severely  bandwidth  limited  autonomous  military  vehicles. 

(4)  The  theoretical  framework  enables  the  derivation  of  optimal  rates  of  convergence  for  a  class 
of  popular  loss  functions,  up  to  a  logarithmic  factor,  for  a  wide  range  of  approximation 
spaces. 

(5)  Highly  efficient  multigrid  methods  have  been  derived  for  implementation  of  the  methodol¬ 
ogy  based  on  techniques  originating  in  multilevel  optimization  and  numerical  methods  for 
partial  differential  equations. 

Further  details  of  this  approach  can  be  found  in  the  references  [23,  24,  8,  25]. 
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4.  Implicit  Geometric  Representations  and  Operations 

During  the  MURI  project,  the  UCI  team  has  focused  on  processing  and  analysis  of  point  clouds 
and  implicit  surface  reconstruction  from  point  clouds  that  can  be  very  useful  for  3D  modeling  of 
terrain  from  LiDAR  measurements.  In  particular  we  have  achieved  the  following: 

•  Robust  and  accurate  point  cloud  normal  estimation,  denoising  and  segmentation. 

•  Efficient  and  robust  implicit  surface  reconstruction  for  point  clouds  based  on  convexified 
image  segmentation. 

•  Analysis  and  understanding  of  point  clouds  using  geometric  partial  differential  equation 


(PDE). 


The  UCI  teams  has  collaborated  with  their  counterparts  at  Texas  A&M  and  the  University  of  South 
Carolina  to  provide  accurate  and  efficient  normal  estimation  for  streaming  surface  reconstruction 
using  wavelets.  Methods  for  point  cloud  segmentation  and  implicit  surface  reconstruction  have 
been  combined  with  visibility  of  point  clouds  and  exploratory  path  planning  algorithms  developed 
by  the  University  of  Texas  and  UCLA.  Below  is  a  brief  summary  of  these  results. 

4.1.  Robust  and  accurate  point  cloud  normal  estimation,  denoising  and  segmentation.  Sur¬ 
face  normal  approximation  is  an  important  but  challenging  task  that  is  utilized  by  several  appli¬ 
cations  including  reconstruction,  estimating  local  feature  size,  computer  aided  design  (CAD),  and 
inside-outside  queries.  The  UCI  team  has  developed  an  improved  surface  normal  approximation 
method,  based  on  recasting  the  popular  principal  component  analysis  formulation  as  a  constrained 
least  squares  problem  in  [16].  The  new  approach  is  robust  to  singularities  in  the  data,  such  as 
intersecting  planes,  and  also  incorporates  data  denoising.  Principal  component  analysis  (PCA)  is 
a  popular  method  for  computing  surface  normal  approximations  from  point  cloud  data.  The  PCA 
normal  approximation  is  accurate  when  the  underlying  surface  is  smooth,  but  tends  to  smear  across 
singularities,  such  as  corners  or  intersecting  planes.  The  smearing  is  caused  by  the  contribution  of 
data  points  from  across  the  singularity.  Figure  13  illustrates  this  phenomenon. 


Figure  13.  PCA  normal  estimation  for  a  cube.  Left:  Standard  PCA.  Notice  the 
smeared  normals  along  the  edges  of  the  cube.  Right:  Nonlinear  least  squares  normal 
estimation. 

The  PCA  normal  approximation  can  also  be  described  as  the  solution  to  the  following  con¬ 
strained  least  squares  problem: 


where  the  rows  of  the  matrix  V  G  rKx3  are  the  difference  vectors  ly  =  —  p.  The  equality  con¬ 

straint  is  necessary  to  ensure  a  nontrivial  solution. 
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In  order  to  reduce  the  contribution  of  points  from  across  the  singularity,  the  approach  incorpo¬ 
rates  an  adaptive  weighting  term  designed  to  deflate  such  contributions.  The  result  is  the  following 
constrained  nonlinear  least  squares  formulation: 


(8) 


n{jn  i 

S.t. 


e  (ajr\) 

lhll  =  i- 


Traditional  weighting  terms  place  emphasis  on  proximity,  and  also  serve  as  a  window  function 
which  isolates  the  local  neighbors.  The  weighting  term  e~^a^  adaptively  deflates  the  contribu¬ 
tion  of  terms  with  high  orthogonality  mismatch  at  a  rate  defined  by  the  parameter  X.  Naturally, 
setting  X  to  zero  results  in  the  original  PC  A  linear  least  squares  problem  7,  with  the  exception  that 
the  difference  vectors  are  taken  from  the  data  point  p  and  not  the  centroid  p.  The  equality  con¬ 
straint  is  easily  absorbed  into  the  objective  function  by  representing  the  unknown  surface  normal  q 
in  spherical  coordinates.  Applying  formulation  8  to  the  box  data  results  in  significant  improvement 
in  the  surface  normal  estimates,  as  illustrated  by  Figure  13. 

Dealing  with  noise  in  LiDAR  data  is  another  important  issue.  Investigators  at  UCI  have  in¬ 
corporated  noise  robustness  into  normal  estimation  in  their  work.  Based  on  robust  and  accurate 
normal  estimation,  they  also  have  developed  a  simple  segmentation  scheme  for  point  clouds  that 
can  decompose  a  point  cloud  into  disconnected  components  and/or  pieces  with  similar  normals  by 
constructing  a  symmetric  adjacency  matrix  for  the  point  cloud.  Figure  14  shows  an  example  of 
segmenting  different  buildings  directly  from  point  clouds. 


Figure  14.  Direct  segmentation  of  point  cloud. 


4.2.  Efficient  and  robust  implicit  surface  reconstruction  for  point  clouds  based  on  convexi- 
fied  image  segmentation.  The  main  advantage  of  using  implicit  representations  is  the  flexibility 
and  robustness  in  dealing  with  complicated  topology  and  the  automatic  representation  of  holes  or 
gaps  in  surface  reconstruction.  Implicit  representations  also  allow  one  to  generate  distance  fields 
which  are  useful  for  estimation  of  the  Hausdorff  metric,  autonomous  navigation,  line  of  sight  cal¬ 
culations  and  other  operations.  The  implicit  representation  can  be  easily  fed  into  implicit  wavelet 
representation  developed  by  MURI  groups  at  University  of  South  Carolina  and  Texas  A&M  Uni¬ 
versity. 

In  the  work  [42]  UCI  investigators  formulate  the  implicit  surface  reconstruction  for  point  clouds 
as  a  three  dimensional  binary  image  segmentation  problem.  In  particular  the  technique  utilizes 
recent  convexified  image  segmentation  models  and  efficient  convex  optimization  techniques  in 
the  formulation.  One  of  the  convexified  image  segmentation  models  the  team  has  adopted  is  the 
TYG-L1  model: 
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(9)  min  f  g(x)\Vu(x)\  +  X\u  —  f\dx. 

In  the  first  term,  g(x)  >  0  is  an  edge  indicator  function  which  is  close  to  zero  at  edge  locations. 
This  term  alone  is  akin  to  the  geodesic  active  contour  model.  For  example,  if  u(x)  is  a  characteristic 
function,  then  /iig(x)|Vn(x)|  is  the  weighted  (by  g(x))  length/area  of  the  boundary  by  the  co-area 
formula  for  total  variation  (TV).  The  second  term  is  a  volumetric  image  fitting  term  in  which  /  is 
the  given  image.  When  g(x)  =  1  and  /  is  a  binary  image,  it  can  be  shown  that  a  global  minimizer  of 

(9)  is  equivalent  to  a  global  minimizer  E  of  the  non-convex  variational  problem  (for  binary  image 
defined  by  Q): 

(10)  nun  |3E|  +X|EA£2|, 

where  A  denoting  the  symmetric  differences  of  the  two  sets.  Another  closely  related  binary  image 
segmentation  model  we  used  is  the  following  CVG  model, 

(11)  min  [  g(x)\Vu(x)\+X[(f-ci)2-(f-C2)2]u(x)dx. 

0<u<l,ci,c2JQ. 

Here,  g(x)  and  fix)  are  exactly  as  in  the  TV-L1  setting,  and  c\  and  C2  are  two  constants. 

However,  different  from  standard  image  segmentation  where  an  initial  image  uniformly  sampled 
on  a  rectangular  grid  is  given,  two  key  issues  are  (1)  there  is  no  initial  image  /  given  on  a  regular 
grid,  and  (2)  a  good  edge  indicator  g(x)  needs  to  be  constructed.  Researchers  in  the  UCI  team  have 
developed  novel  ways  to  construct  this  information  directly  from  the  scattered  point  clouds. 

To  construct  an  initial  image  that  can  provide  a  good  indication  of  inside  and  outside  regions, 
they  use  a  consistent  normal  estimation  developed  earlier  in  this  MURI  project  [16]  and  incorpo¬ 
rates  volumetric  information,  such  as  line  of  sight.  To  construct  a  sharp  edge  indicator  that  can 
provide  a  accurate  data  fitting  term  that  is  robust  to  noise  UCI  uses  a  superposition  of  anisotropic 
Gaussian  kernels  where  the  direction  and  anisotropy  are  defined  based  on  local  principal  compo¬ 
nent  analysis  (PCA).  Below,  a  few  numerical  results  are  provided  that  show  both  efficiency  and 
robustness  of  the  proposed  method  in  the  construction  of  water  tight  surfaces  for  PCOs  with  com¬ 
plicated  topology,  noise,  and  holes.  A  comparison  of  the  method  is  given  with  the  most  popular  and 
state  of  the  art  implicit  surface  reconstruction  algorithm,  Poisson  surface  reconstruction,  proposed 
in  [46], 

Figures  15  and  16  show  the  ability  of  the  method  to  fill  holes  or  gaps  in  the  data.  Figure  17 
shows  the  reconstruction  from  highly  non-uniform  and  sparse  LiDAR  data. 

4.3.  Analysis  and  understanding  of  point  clouds  using  geometric  PDEs.  Point  clouds  provide 
a  basic  and  intrinsic  way  of  sampling  and  representing  geometric  objects  in  3D  modeling  as  well 
as  conveying  information  in  higher  dimensions.  Local  linear  approximations,  such  as  generated 
by  PCA,  can  only  reveal  local  structure.  Understanding  or  characterizing  global  structure  is  a  very 
challenging  task.  By  analyzing  the  properties  of  geometric  intrinsic  differential  operators  one  can 
obtain  global  information  about  the  underlying  geometry.  The  UCI  team  has  developed  a  general 
framework  and  computational  methods  for  solving  geometric  partial  differential  equations  on  point 
clouds  in  3D  and  higher  dimensions  [43, 41,  35].  The  discrete  approximation  of  differential  opera¬ 
tors  on  the  underlying  manifold  represented  by  point  clouds  is  based  only  on  local  reconstruction, 
which  is  simple,  efficient  and  accurate.  In  particular,  using  global  differential  operators,  such  as  the 
Laplace-Beltrami  (LB)  operator,  one  can  link  local  reconstruction  to  a  global  manifold  structure 
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(a)  (b)  (c) 


Figure  15.  (a)  knot  with  gaps,  (b)  is  our  reconstructed  surface,  (c)  is  result  of 
Poisson  surface  reconstruction. 

and  extract  intrinsic  geometric  information  of  the  point  cloud.  The  framework  extends  to  mani¬ 
folds  of  any  (co-)dimension  with  or  without  boundary.  The  complexity  of  our  method  scales  well 
with  the  total  number  of  points  and  the  true  dimension  of  the  manifold,  not  the  embedded  dimen¬ 
sion.  Another  important  application  of  our  method  is  in  data  science,  where  the  task  of  visualizing, 
extracting  information,  analyzing  and  inferring  underlying  structure  from  data  samples  is  ubiqui¬ 
tous.  In  many  cases,  point  cloud  data  resides  or  is  believed  to  reside  on  or  near  a  low-dimensional 
manifold  in  a  much  higher  dimensional  ambient  space.  Although  there  are  useful  tools,  such  as 
the  principal  component  analysis  (PCA)  to  provide  local  dimension  and  linear  approximations  of 
structure,  it  is  very  challenging  to  extract  global  information  and  to  understand  structure  in  general. 
Mathematically  and  computationally  one  can  gain  intrinsic  information  by  studying  the  behavior  of 
differential  equations,  such  as  the  heat  equation  or  the  eigenvalue  problem  for  differential  operators 
such  as  Laplace-Beltrami  operator,  on  manifolds. 

The  framework  developed  by  UCI  in  [43,  41,  35]  solves  PDEs  directly  on  point  clouds  without 
using  parametrization,  triangulation  or  grid,  which  can  be  difficult  to  construct  in  general  and  may 
introduce  artifacts.  The  key  idea  is  that  one  can  define  differential  operators  on  a  manifold  by 
local  construction  of  the  manifold.  In  another  word,  once  we  can  construct  a  function  as  well 
as  the  manifold  locally  in  a  common  reference  coordinate,  we  can  differentiate  the  function  with 
respect  to  the  metric  of  the  underlying  manifold  simply  using  the  chain  rule.  So  in  our  method, 
we  only  need  to  use  the  K-nearest  neighbor  (KNN)  points  to  define  a  local  intrinsic  coordinate 
system  using  PCA  and  to  construct  the  manifold  and  function  locally  using  least  squares.  Local 
approximation  error  and  super-convergence  results  have  been  studied  in  [43]  and  applications  to 
geometric  understanding  of  point  clouds  have  been  presented  in  [41]. 

Here  we  present  experiments  (see  Figures  18-21)  for  the  demonstration  of  computing  heat  ker¬ 
nels,  distance  maps  and  geodesics  on  points  clouds,  computing  eigenvalues  and  eigenfunctions  for 
Laplace-Beltrami  operators,  and  registration  between  point  clouds  using  shape  descriptors  based 
on  eigensystems  of  the  Laplace  operator  that  are  invariant  under  isometric  deformations  and  scal¬ 
ing. 

4.4.  3-D  Nonlocal  Total  Variation  (NLTV)  Methods:  Typical  lidar  systems  transmit  a  laser 
pulse  and  receive  a  deluge  of  data  in  the  form  of  a  point  cloud.  This  point  cloud  contains  im¬ 
portant  information  reflected  from  the  scene  of  interest,  and  can  be  quite  voluminous.  As  a  result 
it  challenges  current  reconstruction  methods  and  storage  capabilities.  Further,  the  sensitivity  of  the 
photo-detectors  (e.g., Geiger-mode  avalanche  photodiodes)  can  generate  significant  noise. 
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(d)  (e) 


Figure  16.  (a)  bunny  with  holes,  (b)  &  (c)  is  our  reconstructed  surface,  (d)  &  (e) 
is  result  of  Poisson  surface  reconstruction. 


Current  state-of-the-art  three-dimensional  surface  reconstruction  algorithms  require  the  use  of  a 
grid  to  determine  measures  of  gradient  and  proximity  of  points.  However,  forcing  data  points  onto 
a  grid  can  result  in  a  loss  of  subtle  geometric  information  inherent  in  the  scene. 

Over  the  past  year,  researchers  at  UCLA  have  initiated  work  on  a  3-D  grid-free,  generalization  of 
regularization  with  non-local  methods,  which  are  already  used  extensively  to  denoise  2-D  images. 
The  technique  examines  3-D  "cloud  patches"  and  tries  to  find  similar  geometric  characteristics.  By 
minimizing  the  nonlocal  total  variation  (NLTV)  between  these  characteristics  the  method  is  able 
to  significantly  denoise  a  3-D  cloud  data  set.  We  briefly  overview  this  method.  Essentially,  the 
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(a)  LiDAR  data  (b)  implicit  surface  reconstruction 
Figure  17 


FIGURE  18.  eigenfunctions  of  Laplace-Beltrami  operator  on  point  clouds 


Figure  19.  Heat  kernel  on  point  cloud 


method  seeks  to  use  the  unique  geometry  of  individual  patch  clouds  to  identify  specific  structures. 
If  p  and  q  denote  two  points  in  the  point  cloud  data,  a  patch  cloud  about  p  or  q  is  defined  in  terms 
of  its  k  nearest  neighbors.  The  characteristic  Cf(p)  :=  p  —  p  is  defined  for  each  patch  cloud  by 
subtracting  the  centroid  p.  It  is  apparent  that  the  characteristic  C/(p)  is  analogous  to  the  intensity 
of  a  pixel  in  a  2-D  image.  After  normalization,  the  geometry  description  Q(p)  is  used  to  describe 
the  geometry  in  the  vicinity  of  the  point  p,  and  a  nonlocal  measure  of  similarity  between  two  patch 
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Figure  20.  Distance  maps  and  geodesics  on  point  clouds 
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Figure  21 .  Registration  of  point  clouds  (with  noise). 


clouds  centered  at  p  and  q  is  given  by  the  weight  function  Wf(p,q)  defined  as 


l  II  £/(?)- ff/MIlj 

Wf(p,q)  ■■=  ^re  "2 


Finally,  denoising  is  achieved  by  solving  either  the  minimization  in  terms  of  the  Lj  norm, 

njn  (j p\Vwu\  +  ^\\u-f\\2 


min 

u 

or  the  minimization  in  terms  of  the  L\  norm, 

min 


M11  ^  ^U^-/lli) 


where  the  operate  Vw  is  defined  as 

(Vwn)  ( p,q )  :=  (Cu(q)  -  Cu{p ))  ^JwJ{p^q) 

The  simulated  images  below  show  a  noisy  scene  on  the  left,  and  the  denoised  version  using  our 
algorithm  in  the  center  and  on  the  right.  Notice  that  changing  codimension  presents  no  difficulties. 
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The  approach  lends  itself  to  the  efficient  design  of  point  cloud  dictionaries  which  can  lead  to 
multiscale  sparse  reconstruction  of  noisy  images  and  has  led  to  truly  remarkable  denoising  in  the 
level  set  representation  of  interpolating  implicit  surfaces. 


Figure  22.  NLTV  Denoising  Example.  Left:  Noised  corrupted  geometry  includ¬ 
ing  filamentary  structure.  Right:  Denoised  geometry  obtained  via  NLTV 


4.5.  Change  Detection  and  Surveillance.  Figure  23  shows  a  single  frame  of  a  video  surveillance 
stream  from  a  parking  lot  at  the  Adelphi  Army  base.  The  upper  left  shows  the  scene  with  40  %  ran¬ 
domly  chosen  missing  points.  The  upper  right  figure  depicts  the  difference  between  two  successive 
frames.  The  lower  left  frame  is  the  result  of  application  of  the  robust  PCA  algorithm  of  [14].  This 
method  uses  L\  theory  to  find  the  sparse  component  (background)  and  the  nuclear  norm  to  find  the 
low  rank  component  (motion).  Our  method,  devised  in  [29],  uses  the  tight  frame  decomposition  of 
[13],  minimizing  the  L\  sum  of  the  coefficients.  These  results  are  extremely  promising  and  suggest 
new  avenues  for  achieving  state-of-the-art  surveillance. 

4.6.  A  Fast  Convex  Optimization  Method  for  Surface  Reconstruction.  In  [28]  we  developed 
a  method  for  distance  preservation  in  level  set  evolution  which  is  very  fast,  using  the  split  Breg- 
man  algorithm  of  Goldstein  and  Osher  [31].  We  reduce  surface  interpolation  to  a  total  variation 
minimization  problem,  as  in  [30]  which  involves  a  redistancing  step.  The  final  results  are  state-of- 
the-art  in  speed  and  are  quite  accurate.  See  Figures  24-27 

In  [36]  we  developed  a  splitting  method  based  again  on  Bregman  iteration  for  optimization 
with  orthogonality  constraints.  This  is  a  nonconvex  problem  but  we  use  analytic  solutions  in  the 
nonconvex  minimization  step.  We  obtain  state-of-the-art  results  for  conformal  mapping  for  surface 
construction,  direction  fields  correction  and  noisy  color  image  restoration.  See  Figures  28-  30. 
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Figure  23.  Video  tracking  with  50%  missing  data:  new  fast  method  allows  track¬ 
ing  Very  noisy  environment. 
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Figure  24.  Proposed  level  set-based  segmentation  method  applied  to  natural  im¬ 
ages.  For  each  result,  we  show  the  segmentation  result  and  the  level  set  function. 
We  plot  the  initial  zero  level  set  in  blue,  the  final  contour  in  pink  and  the  ±1,±2 
level  sets  of  the  final  functions  in  black 
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(a)  grid  120x120x120,  sh  (b)  grid  240x240x240.  391a 


(e)  grid  200t  160x120.  l*la  (d)  grid  490x320x240,  1070a 


(•)  grid  200x  160x120.  1 73b  (f)  grid  490x320x240.  109Ga 


Figure  25.  Reconstructed  surfaces  from  scattered  data  points  at  different  resolutions 


(a)  grid  120x240x120.  173a  (b)  grid  240x480x240.  240a 


(c)  grid  00x00x120.  4Gs  (d)  grid  120x120x240.  31&a 


Figure  26.  Reconstructed  surfaces  from  scattered  data  points  at  different  resolutions 
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Figure  27.  Comparison  of  quality  of  segmentation  and  speed  for  the  different 
methods  on  a  dataset  of  72  images.  Quality  of  the  LSF  is  measured  in  terms  of 
the  penalty  term  at  convergence  ^  /  (| Vcf>|  —  1)  when  the  obtained  contours  are 
equivalent.  Our  method  preserves  the  SDF  almost  as  well  as  redistancing,  clearly 
better  than  Li  et  a/.’s  method  and  is  faster  than  any  of  them. 
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Figure  28.  Comparison  of  the  proposed  algorithm  with  the  curvilinear  algorithm 
and  the  gradient  descent  method.  Surfaces  are  coded  by  the  conformal  factors  ob¬ 
tained  from  the  resulting  conformal  mappings.  Histograms  show  angle  difference 
between  triangles  on  the  input  surface  9V[  and  the  corresponding  triangles  on  the 
obtained  map.  The  last  column  illustrates  the  energy  evolution  via  iteration  num¬ 
bers. 
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Figure  29.  Comparison  between  the  proposed  algorithm  and  the  curvilinear 

search  algorithm.  The  first  column  shows  the  initial  inputs  F  for  both  algorithms. 
The  second  and  the  third  columns  are  the  results  obtained  from  the  curvilinear  al¬ 
gorithm  and  our  proposed  SOC  algorithm,  respectively.  The  last  column  illustrates 
the  energy  evolution  via  the  iteration  numbers. 
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FIGURE  30.  Comparison  between  the  proposed  algorithm  and  the  curvilinear  al¬ 
gorithm  with  fixed  30  iterations.  The  first  column  shows  the  input  images  contam¬ 
inated  by  two  different  levels  of  Gaussian  noise  on  their  chromaticity.  The  second 
and  the  third  columns  are  results  obtained  from  the  curvilinear  algorithms  and  our 
proposed  SOC  algorithm,  respectively.  The  last  column  illustrates  the  energy  evo¬ 
lution  via  iteration  numbers. 


5.  Visibility  of  point  clouds  and  applications  to  path  planning 

PROBLEMS 

The  problem  of  visibility  seeks  to  determine  the  regions  in  space  visible  to  a  given  set  of  ob¬ 
servers  in  the  presence  of  occlusions.  UT  along  with  its  collaborators,  TAMU,  Univ.  South  Car¬ 
olina  and  UCLA,  have  been  developing  algorithms  that  reconstruct  visible  surfaces  of  the  occlud¬ 
ing  objects  based  on  the  point  cloud  data  sampled  from  the  opaque  objects  in  the  environment.  The 
MURI  team  members  have  been  working  on  applying  the  developed  point  cloud  reconstruction  al¬ 
gorithms  to  the  problem  of  exploring  environments  with  complicated  and  unknown  obstacles  using 
single  or  multiple  observers.  As  a  result  of  the  exploration,  a  complete  map  of  the  environment 
is  obtained.  A  convergence  proof  [39]  has  been  provided  indicating  suitability  of  the  proposed  al¬ 
gorithm  for  some  general  types  of  environments.  Various  post-processing  optimization  techniques 
may  be  considered  to  obtain  a  more  uniform  exposure  of  the  environment  along  the  path. 
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In  the  point  cloud  database,  there  are  structures  such  as  overpasses  and  multi-story,  perhaps  ir¬ 
regular,  buildings  that  pose  additional  challenges  for  the  previously  developed  algorithms.  During 
the  final  year  of  this  project,  following  the  feedback  of  Year  4  government  review,  we  have  been 
developing  new  visibility-based  “non-myopic”  exploration  algorithms  for  the  fully  3D  cases.  The 
essential  building  block  consists  of  a  class  of  functionals,  hereafter  referred  to  as  visibility  “met¬ 
rics”,  that  quantifies  the  potential  gain  in  new  visibility  information  by  placing  a  new  observing 
location  at  certain  designated  locations,  given  the  previous  observing  locations. 

Below,  we  briefly  outline  the  our  mathematical  and  algorithmic  approaches.  We  first  list  the 
essential  ingredients  in  defining  such  type  of  metrics: 

•  The  set  of  observing  locations  O:  The  metric  will  depends  on  the  visibility  from  a  set  of 
observing  locations  contained  in  O  =  {xi ,  •  •  •  ,  xy- } . 

•  The  obstacles  Cl:  They  are  assumed  to  be  a  closed  set  containing  finite  number  of  con¬ 
nected  components. 

•  Occlusion  set  S:  The  occlusion  set  contains  points  that  are  not  visible  from  (the  observing 
locations  in)  O.  When  the  visibility  from  O  encompasses  the  entirety  of  the  domain  except 
the  obstacles,  then  the  occlusion  set  is  identical  to  the  obstacles.  In  this  case,  we  shall  say 
that  the  observing  locations  in  O  have  complete  visibility  of  the  domain. 

•  Shadow  boundary  dS:  A  hyper-surface  which  separates  the  domain  locally  into  visible 
and  occluded  regions.  The  shadow  boundary  is  the  attenuation  of  the  visible  portions  of 
the  obstacles  along  the  lines-of- sight.  Note  that  the  visible  portions  of  the  obstacles  are  not 
counted  as  part  of  the  shadow  boundary. 

•  The  portion  of  shadow  boundary  that  is  visible  to  z  will  be  denoted  by  S-- 

•  Viewing  angle  0:  This  is  the  angle  between  the  line-of-sight  that  reaches  the  shadow 
boundary  and  the  normal  of  the  shadow  boundary  at  the  intersection. 

The  viewing  angle  penalizes  viewing  the  shadow  boundary  by  grazing  through  it.  It  also 
reflects  on  the  uncertainties  due  to  errors  in  the  visibility  information.  The  closer  this  angle 
is  to  90°  ,  the  more  sensitive  the  view  will  be  to  perturbation  in  the  obstacles. 

•  Depth  of  occlusion  x:  We  associate  each  point  on  the  shadow  boundary  with  a  depth 
which  reflects  on  the  thickness  of  the  occlusion  set  in  the  direction  that  is  orthogonal  to  the 
shadow  boundary. 

The  depth  value  quantifies  the  volume  of  occlusion  set  that  can  become  visible  if  a  vantage 
point  looks  perpendicularly  at  a  point  on  the  shadow  boundary. 

•  The  weighting  w :  Additional  weighting  may  be  placed  over  the  entire  domain  for  addi¬ 
tional  application-specific  modeling  needs,  such  as  prioritization  of  a  subset  of  the  domain 
or  dealing  with  visibility  with  limited  range. 

With  all  these  ingredients,  we  define  the  metric  as  a  boundary  integral  taking  the  following  form: 
(12)  p(z;0):=  f  w(z,  0)x(y)Q(z,y)dS(y)  =  [  ( wxrz)-ndS(y ). 

JdS-  ./  c)S- 

Here  rz(y)  is  (z  —  y)/\z  —  y\  the  viewing  vector. 

Figure  31  demonstrates  a  visibility  metric  in  2D  following  the  above  formulation.  In  the  left 
subfigure,  the  observing  location  is  shown  by  the  diamond  shape,  the  obstacles  are  the  two  circles 
shown  in  blue.  The  red  curves  show  the  shadow  boundary  and  the  portions  of  the  circles  that  are 
visible  to  the  observing  location.  The  right  subfigure  shows  the  values  of  p(z', (l,0.25)r).  The 
values  increase  from  blue  to  red. 
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Figure  3 1 .  The  values  of  a  visibility  metric  defined  in  (12)  in  a  domain  containing 
two  circles.  The  cross  shows  where  the  metric  is  maximum. 

We  consider  the  problem  of  planning  a  path  through  an  unknown  environment  that  contains 
complicated  geometry;  the  objective  of  such  path  is  to  map  out  the  environment  using  suitable  tech¬ 
nologies  such  as  the  online  multi-resolution  point  cloud  algorithm  that  is  developed  by  this  MURI 
project.  We  solve  this  problem  by  constructing  a  sequence  of  observing  locations,  xo,xi,  •  ■  ■  ,x^, 
and  a  path  that  goes  through  these  locations.  The  k-th  observing  location,  x\,  will  be  chosen  ac¬ 
cording  to  a  visibility  metric,  using  information  obtained  from  the  observing  locations  xo,  •  •  *  jc*—  i . 
Figure  32  demonstrates  a  step-by-step  computer  simulation  using  this  technique.  Figure  33  shows 
a  simulation  result  for  a  more  complicated  set  up.  We  point  out  here  that  this  procedure  is  trans¬ 
parent  to  the  dimension  of  the  problem  -  the  construction  of  the  observing  locations  depend  solely 
on  the  visibility  metric,  which  does  not  require  any  dimension  specific  logic.  Indeed,  the  ini¬ 
tial  3D  implementation  of  the  proposed  visibility  metric  in  our  navigation  strategy  produced  very 
promising  results  for  regular  building-like  structures  as  shown  in  Figure  34,  where  we  performed 
simulations  using  four  buildings. 

Furthermore,  it  is  conceivable  that  in  certain  applications,  one  is  interested  in  mapping  out  cer¬ 
tain  part  of  the  domain  sooner  than  the  other  parts.  The  goal  to  “put  some  region  under  surveillance 
sooner”  translates  into  the  objective  of  having  visibility  on  that  region  using  as  few  observing  lo¬ 
cations  as  possible,  starting  from  xq-  Thus,  the  subscript  in  the  observing  location  relates  to  the 
progression  of  time  as  well. 

During  the  final  period  of  the  MURI  project,  we  have  worked  on  reducing  the  computational 
complexity  of  the  new  approach  and  collaborated  rather  closely  with  the  Texas  A&M  team  to 
interface  the  new  visibility  algorithm  with  their  point  cloud  learning  code  as  well  as  testing  the  in¬ 
tegrated  code  in  the  USC  MOUT  simulation  site;  in  the  MURI  Final  Review,  we  had  demonstrated 
a  satisfactory  performance  of  the  integrated  code  in  autonomous  and  full  3D  exploration  using  the 
simulated  MOUT  site. 

In  addition,  during  this  last  period  of  the  MURI  project,  we  started  to  analyze  the  performance 
of  the  new  visibility  algorithm  -  to  obtain  some  fundamental  understanding  on  how  many  vantage 
points,  with  respect  to  the  geometrical  and  topological  complexity  of  the  obstacles  in  the  scene, 
will  be  generated  by  the  new  algorithm  in  order  to  obtain  complete  visual  surveillance  of  the 
scene  (i.e.  no  blind  spots),  and  if  there  were  a  known  optimal  algorithm,  how  does  the  current 
solution  compare  with  the  optimal  solution?  In  Figure  35,  we  show  the  vantage  point  locations  as 
planned  by  our  algorithm  for  complete  surveillance  of  two  and  four  spheres  in  a  scene.  Our  many 
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Figure  32.  Generation  of  an  “optimized”  set  of  observing  locations  via  the  use 
of  a  visibility  metric  /u  defined  in  (12).  The  blue  curves  shown  in  the  top  row  are 
the  estimates  of  the  obstacles  obtained  from  the  observing  locations.  The  crosses 
correspond  to  the  newly  added  observing  location.  The  diamonds  indicate  the  ob¬ 
serving  locations.  The  images  on  the  second  row  show  the  values  of  the  visibility 
metric  so  that  the  whiter  the  color  at  a  point,  the  larger  the  value  is. 


Figure  33.  A  simulation  involving  online-visibility  estimates  for  mapping  out  the 
obstacles  in  the  domain.  The  plot  on  the  left  shows  the  constructed  observing  loca¬ 
tions  in  diamond  shape  and  the  obstacle  estimates  as  well  as  the  shadow  boundaries 
at  different  stages.  The  plot  on  the  right  shows  the  estimated  obstacles. 

experiments  suggest  that  with  each  additional  sphere  in  the  scene,  only  1  or  2  additional  vantage 
points  are  required,  and  this  type  of  performance  seems  to  be  stable  with  respect  to  perturbation 
in  the  sphere  sizes  and  spatial  arrangements.  In  Figures  36  and  37  we  show  similar  efficiency  in 
vantage  point  placement  for  more  complicated  pipeline  (topological)  configurations. 

This  particular  line  of  research  was  a  new  initiative  of  our  MURI  project  and  the  strong  potential 
which  was  exposed,  has  enabled  the  further  development  of  this  research  through  additional  ARO 
support. 
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FIGURE  34.  A  simulation  using  the  new  visibility  metric  based  algorithm  in  a  3D 
domain.  Note  that  the  green  surfaces  are  the  intersections  of  all  the  shadow  bound¬ 
aries,  and  when  complete  visibility  of  the  free  space  is  obtained  by  the  vantage 
points,  the  green  surface  will  correspond  to  the  obstacles’  surface. 


Figure  35 .  The  performance  of  the  new  visibility  algorithm  for  exploring  two  and 
four  spheres.  The  left  plot  shows  the  decay  of  the  energy  additional  vantage  points 
are  being  added.  Notice  that  the  number  of  vantage  points  needed  seem  to  grow 
linearly  with  the  number  of  spheres  in  the  scene. 

6.  Verification  and  Data  Acquisition 

The  development  and  testing  of  algorithms  and  software  for  terrain  rendering  is  very  dependent 
on  having  robust  data  sets.  Unfortunately,  the  project  has  been  unable  to  obtain  sufficient  satis¬ 
factory  data  sets  which  can  be  used  demonstration  of  our  algorithm  advances.  In  order  to  address 
this  deficiency,  the  project  has  taken  two  steps:  (i)  develop  simulators  for  realistic  LIDAR  data 
generation,  (ii)  generate  the  project’s  own  data  sets  and  test  its  software  at  available  test  sites. 

6.1.  Enhanced  USC  Simulator  Capability.  Building  upon  a  previous  collaboration  with  VT 
project  personnel,  USC  enhanced  the  capability  of  its  simulator  of  a  virtual  environment  of  the 
Ft.  Benning  MOUT  site.  The  earlier  version  was  designed  to  generate  flight  video  which  was  used 
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Figure  36.  The  performance  of  the  new  visibility  algorithm  for  exploring  a  simple 
3D  pipeline  configuration.  The  right  plot  shows  the  decay  of  the  energy  additional 
vantage  points  are  being  added. 


Figure  37.  The  performance  of  the  new  visibility  algorithm  for  exploring  a 
slightly  more  complex  3D  pipeline  configuration.  The  right  plot  shows  the  decay 
of  the  energy  additional  vantage  points  are  being  added.  Notice  that  the  number 
of  vantage  points  needed  seem  to  grow  linearly  with  the  number  of  spheres  in  the 
scene.  Notice  also  in  the  left  subfigure  the  symmetry  of  the  planned  vantage  point 
locations. 

to  develop  autonomous  navigation  algorithms  using  vision-based  Structure  From  Motion  (SFM) 
point  clouds.  In  years  one  through  three  of  the  project  the  capabilities  of  the  simulator  were  up¬ 
graded  and  enhanced  to  include  additional  sensor  models  and  capabilities,  including  lidar.  These 
improvements  also  included  an  improved  interface  that  allowed  for  direct  control  of  the  simulator 
via  Matlab  as  well  as  remote  control  over  a  local  area  network.  The  simulator  allows  the  virtual 
deployment  of  multiple  mobile  platforms  (ground  and  airborne)  to  generate  sensor  data  within  the 
virtual  environment.  The  textured  geometric  data  base  originally  used  Open-Flight  data  structures 
and  has  been  extended  to  incorporate  several  more  popular  3D  and  texture  formats.  The  added 
support  of  additional  formats  allows  for  existing  assets  typically  created  for  high-end  games  to 
be  retasked  for  the  ARL-suggested  surveillance  and  change-detection  experiments  among  other 
simulated  exercises.  See  Figure  38. 
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a)  b) 


Figure  38.  Project  Simulation  capabilities:  a)  Multiple  vehicle  sensor  pack¬ 
ages  (purple  frustrum  for  ground  vehicle,  yellow  for  air)  moving  along  designated 
paths,  b)  lidar  simulated  point  cloud  acquired  along  flight  path  (colored  by  height). 


In  years  four  and  five  the  capabilities  of  the  simulator  were  extended  with  various  model  sensors, 
including  SICK,  PILAR  and  Flash  Lidar,  which  may  be  tested  for  real-time  analysis  of  generated 
point  clouds,  design  of  sensor  packages,  and  testing  of  collaborative  dynamic  sensor  positioning. 
Additionally,  external  packages  were  updated  to  augment  the  simulator  for  applications  such  as 
feature  tracking,  SFM  and  point  cloud  editing.  The  in-house  rendering  engine  used  by  the  simu¬ 
lator  was  also  updated  to  use  the  acceleration  structures  found  in  OpenSceneGraph.  This  update 
reduced  rendering  times  and  allowed  for  both  lidar  and  camera  sensors  to  share  data  structures 
used  during  simulated  data  acquisition. 

In  order  to  further  develop  one  of  the  project’s  designated  demonstration  application  areas,  the 
simulator  has  been  coupled  with  MAV  flight  control  modules  created  using  Simulink  and  Matlab 
in  order  to  design  and  test  autonomous  navigation  algorithms  using  MURI  surfaces  constructed 
from  vision  and  lidar  sensors.  Moreover,  the  simulator  has  been  designed  to  easily  accommodate 
and  visualize  line-of-sight  algorithms  for  MURI  constructed  surfaces. 

As  part  of  a  collaboration  with  TAMU,  an  effort  began  late  in  year  five  to  replace  the  in-house 
engine  with  Unity,  a  cross-platform  and  well  supported  game  engine.  The  inclusion  of  the  new 
engine  will  allow  simulated  scenes  to  contain  higher  polygon  counts  and  thus  provide  more  realistic 
simulated  sensor  data.  Along  with  this  update  to  the  software,  efforts  are  underway  to  update  and 
improve  the  realism  of  the  individual  assets,  including  more  realistic  and  higher-polygon  flora. 

6.2.  Experimental  Verification.  The  MURI  team  places  a  high  value  on  experimental  verifica¬ 
tion  and  validation  of  the  theory,  algorithms  and  software  developed  in  this  research  program. 
Experimental  field  testing  of  the  research  in  this  MURI  program  prepares  the  team  in  providing  di¬ 
rect,  pertinent  research  to  DoD  laboratories.  It  should  be  emphasized  that  the  MURI  has  supported 
the  researchers  and  their  assistants  in  their  development  of  parts  of  the  testbed,  but  the  hardware 
components  of  the  testbed  itself  are  assets  of  the  Unmanned  System  Laboratory  at  Virginia  Tech. 
Virginia  Tech  has  extended  its  techniques  for  creating  dynamic  point  cloud  measurement  from  ex¬ 
periment  to  include  point  cloud  data  from  two  rotationally  articulating  LIDAR  line  scanners.  One 
drawback  of  the  previous  configuration  described  in  the  last  report  was  that  LIDAR  data  collection 
was  restricted  to  fixed  orientation  with  respect  to  the  experimental  vehicle.  This  meant  that  data  ac¬ 
quisition  was  limited  to  only  certain  directions  relative  to  the  vehicle  trajectory  for  one  experiment. 
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FIGURE  39.  a)  Year  1  Virginia  Tech  ground  vehicle  equipped  with  a  six  sensor 
package  which  includes  2  horizontal  line-scanning  lidar,  2  vertical  line-scanning 
lidar,  stereo  video,  and  an  integrated  IMU/GPS  unit,  b)  Four  channels  of  raw  lidar 
data,  colored-tagged  by  the  appropriate  lidar  sensor,  and  coregistered. 


A  great  deal  of  time  and  effort  was  required  to  construct  the  vehicle  trajectory  that  would  have  suf¬ 
ficient  point  cloud  density.  The  Virginia  Tech  testbed  now  can  provide  dynamically  collected  data 
sets  to  the  MURI  team  members  of  a  much  higher  density,  which  was  a  significant  drawback  to 
the  earlier  efforts.  Now  the  Unmanned  Systems  Laboratory  at  VTech  is  capable  of  collecting  dy¬ 
namic  data  sets  from  a  system  that  is  configurable  with  a  subset  of  (1)  two  stereo  vision  cameras 
operating  at  a  maximum  resolution  of  640x480  at  60hz  frame  rate,  (2)  two  channels  of  80  meter 
range,  single-line-scan  LIDAR  units,  (3)  two  channels  of  30  meter  range  single-linescan,  (4)  two 
articulating  rotating  LIDAR  units,  and/or  an  (4)  IMU/GPS.  South  Carolina  researchers  have  con¬ 
tinued  to  work  in  a  support  role  to  the  VTech  team’s  data  collection  efforts  by  (i)  collaborating  in 
the  design  and  coding  of  the  Application  Programmers  Interfaces  (API)  for  hardware  and  software 
integration  of  real-time  data  assimilation  from  the  sensors,  (ii)  providing  a  real-time  on-board  3D 
renderer  of  the  scene  for  initial  testing. 

7.  Technology  Transfer:  Transitioning  and  outreach 

The  success  of  this  project  will  be  measured  by  the  usefulness  of  the  technology  that  it  develops. 
This  is  dependent  on  the  project  having  significant  recognition  in  the  defense  community  and 
having  specific  transitioning  projects. 

7.1.  Transitioning.  The  project  has  made  significant  efforts  for  transitioning  its  technology  to  the 
defense  sector.  Through  a  number  of  transition/evaluation  opportunities  identified  by  the  ARO 
Program,  project  members  have  interacted  with  researchers  from  ARL-Adelphi,  AFRL-WP,  de¬ 
fense  industry  contractors,  and  NATO  SET- 118  appointed  members  in  order  to  gain  insight  into 
relevant  DOD  problems,  current  methods  of  approach,  and  the  obstacles  to  meeting  defense  needs. 
This  effort  helped  guide  the  project’s  formulation  of  application  areas  and  challenge  problems. 

7.1.1.  Formal  Industrial  Collaborations:  SBIR/STTR  projects.  Project  members  were  directly  in¬ 
volved  in  technology  transfer  through  funded  Phase  II  SBIR  and  STTR  awards.  South  Carolina 
personnel  (Sharpley,  DeVore,  Hielsberg  and  Binev)  teamed  with  Radiance  Technologies  (Andrew 
Thies,  PI)  as  its  industrial  partner  to  provide  geometric  mapping  and  navigation  capabilities  to  the 
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war  fighter  in  an  ARO-STTR  entitled  “Software  for  Generating  Geometrically  and  Topologically 
Accurate  Urban  Terrain  Models  Using  Implicit  Methods”.  Further,  Radiance  (Thies,  PI)  teamed 
with  South  Carolina  (Sharpley)  and  Virginia  Tech  (Kurdila)  to  work  on  an  AFRL-funded  SBIR 
Phase  II  project  entitled  “Innovative  Micro  Air  Vehicles  &  Control  Techniques  for  Urban  Environ¬ 
ments”.  The  objective  of  the  project  was  to  provide  surveillance,  mapping,  autonomous  navigation 
and  control  capabilities  for  micro-aerial  vehicles  in  real-time.  These  projects  were  successfully 
completed  in  March  2010  and  November  2011,  respectively. 

7.1.2.  Sponsored  International  Defense  Collaboration:  NATO  Research  Task  Group  SET-118.  A 
MURI  project  member  (Sharpley)  was  appointed  to  a  three  year  term  as  a  US  representative  to 
the  NATO  Research  and  Technology  Organization  (RTO)  Research  Task  Group  (RTG)  on  03D- 
Modelling  of  Urban  TerrainO  (SET-118).  Briefly,  this  task  group  was  formed  to  study  various 
representation  forms  for  objects  in  urban  terrain,  to  inspect  procedures  for  automatic  scene  recon¬ 
structions  by  exploiting  the  data  of  modern  sensors,  and  to  discuss  potential  assessment  metrics 
and  criteria,  as  well  as  interoperability  in  terrain  databases.  The  sensor  technologies  investigated 
covered  active  and  passive  sensors,  such  as  flash  laser,  video,  and  interferometric  SAR.  A  fuller 
discussion  of  the  relevance  of  3D  models  and  military  applications  which  benefit  from  the  exis¬ 
tence  of  3D  information,  were  itemized  and  discussed  in  its  final  report  [15]. 

7.1.3.  Army  Research  Lab  Collaboration:  ARL  Adelphi.  A  student  at  UCLA,  Zhaohui  Guo,  vis¬ 
ited  the  ARL  Adelphi  Lab  for  3  weeks  and  worked  with  Nasser  Nasrabadi’s  group  on  Change 
Detection  and  Surveillance,  in  particular  on  DR/FR/Mines  data  sets.  He  introduced  fused  lasso 
together  with  L\  template  matching.  This  improved  existing  hyperspectral  target  detection  algo¬ 
rithms.  The  investigators  have  also  considered  combining  this  with  dictionary  based  approaches. 
A  prior  rank,  intensity,  and  sparsity  models,  based  on  robust  PCA  and  tight  frame  regularization 
methods  has  also  been  developed.  It  has  been  applied  very  successfully  to  track  objects  in  a  noisy 
video  at  the  ARL  Adelphi  Laboratory  parking  lot. 

Figure  23  shows  a  single  frame  of  a  video  surveillance  stream  from  a  parking  lot  at  the  Adelphi 
Army  base.  The  upper  left  shows  the  scene  with  40%  randomly  chosen  missing  points.  The  upper 
right  figure  depicts  the  difference  between  two  successive  frames.  The  lower  left  frame  is  the 
result  of  application  of  the  robust  PCA  algorithm  of  [14].  This  method  uses  L\  theory  to  find  the 
sparse  component  (background)  and  the  nuclear  norm  to  find  the  low  rank  component  (motion). 
Our  method,  devised  in  [29]  uses  the  tight  frame  decomposition  of  [13],  minimizing  the  L\  sum 
of  the  coefficients.  These  results  are  extremely  promising  and  suggest  new  avenues  for  achieving 
state-of-the-art  surveillance.  One  potential  and  promising  approach  is  the  possibility  of  combining 
the  methodology  with  the  fast  interpolation  of  point  clouds  of  [28]  to  do  point  cloud  separation, 
(including  motion)  into  a  low  rank  and  sparse  decompositions. 

7.2.  Data  Security  and  Control.  Raw  data  sets,  manuscripts  and  formal  powerpoint  presenta¬ 
tions  were  provided  by  NATO  SET-118  member  organizations  to  the  South  Carolina  component 
of  the  MURI  for  use  in  Urban  Terrain  Research.  These  research  materials  were  provided  under 
a  FOUO  classification  (“For  Official  Use  Only”)  and  under  the  agreement  that  SET- 118  data  col¬ 
lections  would  be  accessed  only  by  NATO  citizens,  unless  permitted  explicitly  in  writing  by  the 
NATO  partner  providing  a  data  collection.  The  South  Carolina  component  has  complied  with  this 
agreement  and  has  safely  secured  all  materials.  Four  individuals  from  South  Carolina  have  had 
access  to  this  data  (R.  Sharpley,  M.  Hielsberg,  S.  Johnson,  and  J.  Winders)  and  each  has  signed  a 
technology  control  form  specifically  written  for  this  project.  The  Form  (see  attachment  Be  sure 
to  ! insert!)  states  that  the  person  who  is  granted  access  to  the  data  acknowledges  that  technology 
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restrictions  apply  to  non-NATO  citizens,  and  the  requirement  to  formally  document  and  track  any 
transfer  of  sensitive  technology.  The  signed  technology  control  forms  have  been  kept  in  a  secured 
IMI  Firebox,  along  with  original  storage  media  containing  the  NATO  materials  which  were  pro¬ 
vided  by  the  NATO  SET-118  leadership.  Digital  copies  of  SET-118  materials  are  also  kept  on  a 
secure  server  to  which  only  the  four  individuals  had  secure  login  access.  The  server  is  able  to  be 
accessed  only  through  a  secure  local  gigabit  network.  The  firebox  was  kept  in  a  locked  file  cabinet 
(in  the  IMI  laboratory),  to  which  only  M.  Hielsberg  and  R.  Sharpley  had  keys.  Laboratory  doors 
were  locked  at  all  times. 

7.3.  Project  Management  and  Outreach.  The  South  Carolina  team  was  responsible  for  the  over¬ 
all  organization  and  coordination  of  the  project  during  the  first  three  years  of  the  project.  Manage¬ 
ment  during  the  out-year  extension  (Years  4-5)  passed  to  the  Virginia  Tech  team. 

A  website  (http://imi.cas.sc.edu/MURIwebsite)  is  maintained  to  facilitate  collaboration  and  to 
promote  the  MURI  findings  and  activities.  In  addition  to  describing  on-going  activities,  the  website 
provides  Data  and  Code  repositories  available  to  project  members  via  secure  logins.  In  particular 
the  USC  simulator  has  been  made  available  to  all  the  MURI  participants.  The  data  which  has 
been  developed  and  collected  is  used  to  test  algorithms  and  includes  both  simulated  and  real  point 
clouds  derived  from  vision,  lidar  and  other  sensors. 
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